Oct 30 23:55:56.779038 ip-10-0-129-70 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 23:55:56.779050 ip-10-0-129-70 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 23:55:56.779057 ip-10-0-129-70 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 23:55:56.779343 ip-10-0-129-70 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 23:56:06.881895 ip-10-0-129-70 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 23:56:06.881908 ip-10-0-129-70 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot c8f29756d4ce43e889a10ba0a1f6f014 -- Oct 30 23:58:12.572244 ip-10-0-129-70 systemd[1]: Starting Kubernetes Kubelet... Oct 30 23:58:12.932038 ip-10-0-129-70 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:58:12.932038 ip-10-0-129-70 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 23:58:12.932038 ip-10-0-129-70 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:58:12.932648 ip-10-0-129-70 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:58:12.935197 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935120 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:58:12.935197 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935197 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:58:12.935266 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935201 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:58:12.935266 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935205 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:58:12.935266 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935208 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:58:12.935266 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935210 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:58:12.935266 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935213 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:58:12.935266 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935217 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:58:12.935266 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935219 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:58:12.935266 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935222 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:58:12.935266 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935225 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:58:12.935266 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935227 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:58:12.935266 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935231 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:58:12.935266 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935236 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:58:12.935266 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935239 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:58:12.935266 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935242 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:58:12.935266 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935245 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:58:12.935266 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935248 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:58:12.935266 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935251 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:58:12.935266 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935254 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:58:12.935266 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935257 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:58:12.935696 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935259 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:58:12.935696 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935262 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:58:12.935696 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935265 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:58:12.935696 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935267 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:58:12.935696 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935270 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:58:12.935696 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935273 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:58:12.935696 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935276 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:58:12.935696 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935278 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:58:12.935696 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935289 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:58:12.935696 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935293 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:58:12.935696 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935295 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:58:12.935696 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935298 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:58:12.935696 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935300 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:58:12.935696 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935303 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:58:12.935696 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935306 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:58:12.935696 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935308 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:58:12.935696 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935311 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:58:12.935696 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935313 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:58:12.935696 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935316 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:58:12.935696 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935318 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:58:12.936182 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935321 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:58:12.936182 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935323 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:58:12.936182 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935326 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:58:12.936182 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935328 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:58:12.936182 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935331 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:58:12.936182 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935333 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:58:12.936182 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935336 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:58:12.936182 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935340 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:58:12.936182 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935344 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:58:12.936182 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935346 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:58:12.936182 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935349 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:58:12.936182 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935352 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:58:12.936182 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935354 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:58:12.936182 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935358 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:58:12.936182 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935360 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:58:12.936182 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935363 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:58:12.936182 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935365 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:58:12.936182 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935368 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:58:12.936182 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935371 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:58:12.936182 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935374 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:58:12.936673 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935376 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:58:12.936673 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935379 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:58:12.936673 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935384 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:58:12.936673 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935388 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:58:12.936673 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935391 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:58:12.936673 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935394 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:58:12.936673 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935397 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:58:12.936673 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935399 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:58:12.936673 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935402 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:58:12.936673 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935405 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:58:12.936673 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935407 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:58:12.936673 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935410 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:58:12.936673 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935413 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:58:12.936673 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935416 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:58:12.936673 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935418 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:58:12.936673 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935421 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:58:12.936673 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935423 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:58:12.936673 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935426 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:58:12.936673 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935430 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:58:12.937120 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935433 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:58:12.937120 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935436 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:58:12.937120 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935439 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:58:12.937120 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935442 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:58:12.937120 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935444 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:58:12.937120 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935447 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:58:12.937120 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935450 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:58:12.937120 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935453 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:58:12.937120 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935456 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:58:12.937120 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935459 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:58:12.937120 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935461 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:58:12.937120 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935464 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:58:12.937120 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935467 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:58:12.937120 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935469 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:58:12.937120 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935472 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:58:12.937120 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935475 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:58:12.937120 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935477 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:58:12.937120 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935481 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:58:12.937120 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935484 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:58:12.937573 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935487 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:58:12.937573 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935489 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:58:12.937573 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935492 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:58:12.937573 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935494 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:58:12.937573 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935497 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:58:12.937573 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935506 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:58:12.937573 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.935509 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:58:12.937573 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937107 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:58:12.937573 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937116 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:58:12.937573 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937120 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:58:12.937573 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937123 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:58:12.937573 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937126 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:58:12.937573 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937130 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:58:12.937573 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937133 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:58:12.937573 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937136 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:58:12.937573 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937139 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:58:12.937573 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937142 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:58:12.937573 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937144 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:58:12.937573 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937147 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:58:12.937573 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937150 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:58:12.938095 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937152 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:58:12.938095 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937155 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:58:12.938095 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937157 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:58:12.938095 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937161 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:58:12.938095 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937163 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:58:12.938095 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937166 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:58:12.938095 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937168 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:58:12.938095 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937171 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:58:12.938095 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937173 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:58:12.938095 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937176 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:58:12.938095 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937178 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:58:12.938095 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937181 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:58:12.938095 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937184 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:58:12.938095 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937188 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:58:12.938095 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937192 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:58:12.938095 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937195 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:58:12.938095 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937198 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:58:12.938095 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937200 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:58:12.938095 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937203 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:58:12.938556 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937206 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:58:12.938556 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937208 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:58:12.938556 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937211 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:58:12.938556 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937213 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:58:12.938556 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937216 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:58:12.938556 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937218 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:58:12.938556 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937221 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:58:12.938556 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937223 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:58:12.938556 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937226 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:58:12.938556 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937229 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:58:12.938556 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937232 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:58:12.938556 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937234 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:58:12.938556 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937237 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:58:12.938556 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937239 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:58:12.938556 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937241 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:58:12.938556 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937244 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:58:12.938556 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937247 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:58:12.938556 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937250 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:58:12.938556 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937252 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:58:12.938556 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937255 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:58:12.939039 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937257 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:58:12.939039 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937260 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:58:12.939039 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937263 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:58:12.939039 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937266 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:58:12.939039 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937268 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:58:12.939039 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937271 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:58:12.939039 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937274 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:58:12.939039 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937277 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:58:12.939039 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937279 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:58:12.939039 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937282 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:58:12.939039 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937285 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:58:12.939039 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937288 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:58:12.939039 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937291 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:58:12.939039 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937293 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:58:12.939039 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937296 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:58:12.939039 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937298 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:58:12.939039 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937301 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:58:12.939039 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937303 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:58:12.939039 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937306 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:58:12.939506 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937308 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:58:12.939506 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937311 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:58:12.939506 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937313 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:58:12.939506 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937317 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:58:12.939506 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937319 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:58:12.939506 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937322 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:58:12.939506 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937324 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:58:12.939506 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937327 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:58:12.939506 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937329 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:58:12.939506 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937332 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:58:12.939506 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937335 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:58:12.939506 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937337 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:58:12.939506 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937340 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:58:12.939506 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937342 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:58:12.939506 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937347 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:58:12.939506 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937351 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:58:12.939506 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937353 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:58:12.939506 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937356 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:58:12.939506 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937359 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:58:12.939968 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937361 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:58:12.939968 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937364 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:58:12.939968 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937366 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:58:12.939968 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937369 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:58:12.939968 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937372 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:58:12.939968 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937375 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:58:12.939968 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937377 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:58:12.939968 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937380 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:58:12.939968 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937382 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:58:12.939968 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937385 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:58:12.939968 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937387 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:58:12.939968 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937390 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:58:12.939968 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937392 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:58:12.939968 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937395 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:58:12.939968 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937397 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:58:12.939968 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937400 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:58:12.939968 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937872 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:58:12.939968 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937877 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:58:12.939968 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937880 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:58:12.939968 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937882 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:58:12.940452 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937885 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:58:12.940452 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937888 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:58:12.940452 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937891 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:58:12.940452 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937894 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:58:12.940452 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937897 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:58:12.940452 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937900 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:58:12.940452 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937903 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:58:12.940452 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937906 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:58:12.940452 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937910 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:58:12.940452 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937914 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:58:12.940452 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937918 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:58:12.940452 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937920 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:58:12.940452 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937924 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:58:12.940452 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937926 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:58:12.940452 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937929 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:58:12.940452 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937931 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:58:12.940452 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937934 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:58:12.940452 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937937 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:58:12.940452 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937939 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:58:12.940452 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937942 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:58:12.940916 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937944 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:58:12.940916 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937947 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:58:12.940916 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937950 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:58:12.940916 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937979 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:58:12.940916 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937989 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:58:12.940916 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937992 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:58:12.940916 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.937998 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:58:12.940916 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938001 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:58:12.940916 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938004 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:58:12.940916 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938007 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:58:12.940916 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938009 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:58:12.940916 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938012 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:58:12.940916 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938015 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:58:12.940916 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938018 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:58:12.940916 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938021 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:58:12.940916 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938023 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:58:12.940916 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938026 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:58:12.940916 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938028 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:58:12.940916 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938031 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:58:12.941392 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938034 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:58:12.941392 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938036 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:58:12.941392 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938039 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:58:12.941392 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938041 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:58:12.941392 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938044 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:58:12.941392 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938046 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:58:12.941392 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938049 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:58:12.941392 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938052 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:58:12.941392 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938055 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:58:12.941392 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938057 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:58:12.941392 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938060 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:58:12.941392 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938062 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:58:12.941392 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938065 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:58:12.941392 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938068 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:58:12.941392 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938071 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:58:12.941392 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938073 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:58:12.941392 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938075 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:58:12.941392 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938078 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:58:12.941392 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938080 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:58:12.941392 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938085 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:58:12.941937 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938088 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:58:12.941937 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938090 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:58:12.941937 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938092 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:58:12.941937 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938096 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:58:12.941937 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938099 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:58:12.941937 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938101 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:58:12.941937 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938104 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:58:12.941937 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938107 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:58:12.941937 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938110 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:58:12.941937 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938112 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:58:12.941937 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938115 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:58:12.941937 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938117 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:58:12.941937 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938120 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:58:12.941937 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938123 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:58:12.941937 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938125 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:58:12.941937 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938128 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:58:12.941937 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938130 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:58:12.941937 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938133 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:58:12.941937 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938135 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:58:12.941937 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938138 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:58:12.942427 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938141 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:58:12.942427 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938143 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:58:12.942427 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938146 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:58:12.942427 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938149 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:58:12.942427 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938151 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:58:12.942427 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938154 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:58:12.942427 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938156 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:58:12.942427 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938159 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:58:12.942427 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938161 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:58:12.942427 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938164 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:58:12.942427 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938167 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:58:12.942427 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938171 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:58:12.942427 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938173 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:58:12.942427 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938176 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:58:12.942427 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938179 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:58:12.942427 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938181 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:58:12.942427 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938184 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:58:12.942427 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938187 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:58:12.942427 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938189 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:58:12.942427 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938192 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:58:12.942876 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938195 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:58:12.942876 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938200 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:58:12.942876 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.938204 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:58:12.942876 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939068 2574 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 23:58:12.942876 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939079 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 23:58:12.942876 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939087 2574 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 23:58:12.942876 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939092 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 23:58:12.942876 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939097 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 23:58:12.942876 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939100 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 23:58:12.942876 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939105 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 23:58:12.942876 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939109 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 23:58:12.942876 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939112 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 23:58:12.942876 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939115 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 23:58:12.942876 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939119 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 23:58:12.942876 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939123 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 23:58:12.942876 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939126 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 23:58:12.942876 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939129 2574 flags.go:64] FLAG: --cgroup-root="" Oct 30 23:58:12.942876 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939132 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 23:58:12.942876 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939135 2574 flags.go:64] FLAG: --client-ca-file="" Oct 30 23:58:12.942876 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939138 2574 flags.go:64] FLAG: --cloud-provider="external" Oct 30 23:58:12.942876 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939141 2574 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 23:58:12.942876 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939145 2574 flags.go:64] FLAG: --cluster-domain="" Oct 30 23:58:12.942876 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939148 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 23:58:12.943451 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939151 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 23:58:12.943451 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939155 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 23:58:12.943451 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939158 2574 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 23:58:12.943451 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939162 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 23:58:12.943451 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939165 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 23:58:12.943451 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939168 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 23:58:12.943451 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939173 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 23:58:12.943451 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939176 2574 flags.go:64] FLAG: --contention-profiling="false" Oct 30 23:58:12.943451 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939179 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 23:58:12.943451 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939182 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 23:58:12.943451 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939185 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 23:58:12.943451 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939188 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 23:58:12.943451 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939193 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 23:58:12.943451 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939196 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 23:58:12.943451 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939199 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 23:58:12.943451 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939202 2574 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 23:58:12.943451 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939205 2574 flags.go:64] FLAG: --enable-server="true" Oct 30 23:58:12.943451 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939210 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 23:58:12.943451 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939214 2574 flags.go:64] FLAG: --event-burst="100" Oct 30 23:58:12.943451 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939218 2574 flags.go:64] FLAG: --event-qps="50" Oct 30 23:58:12.943451 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939221 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 23:58:12.943451 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939224 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 23:58:12.943451 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939227 2574 flags.go:64] FLAG: --eviction-hard="" Oct 30 23:58:12.943451 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939231 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 23:58:12.944085 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939235 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 23:58:12.944085 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939238 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 23:58:12.944085 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939241 2574 flags.go:64] FLAG: --eviction-soft="" Oct 30 23:58:12.944085 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939244 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 23:58:12.944085 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939247 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 23:58:12.944085 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939250 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 23:58:12.944085 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939253 2574 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 23:58:12.944085 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939256 2574 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 23:58:12.944085 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939259 2574 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 23:58:12.944085 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939262 2574 flags.go:64] FLAG: --feature-gates="" Oct 30 23:58:12.944085 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939269 2574 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 23:58:12.944085 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939272 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 23:58:12.944085 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939275 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 23:58:12.944085 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939278 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 23:58:12.944085 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939282 2574 flags.go:64] FLAG: --healthz-port="10248" Oct 30 23:58:12.944085 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939285 2574 flags.go:64] FLAG: --help="false" Oct 30 23:58:12.944085 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939289 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:58:12.944085 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939292 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 23:58:12.944085 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939295 2574 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 23:58:12.944085 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939298 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 23:58:12.944085 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939301 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 23:58:12.944085 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939305 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 23:58:12.944085 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939308 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 23:58:12.944640 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939311 2574 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 23:58:12.944640 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939314 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 23:58:12.944640 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939318 2574 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 23:58:12.944640 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939321 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 23:58:12.944640 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939324 2574 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 23:58:12.944640 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939327 2574 flags.go:64] FLAG: --kube-reserved="" Oct 30 23:58:12.944640 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939330 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 23:58:12.944640 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939333 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 23:58:12.944640 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939336 2574 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 23:58:12.944640 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939339 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 23:58:12.944640 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939342 2574 flags.go:64] FLAG: --lock-file="" Oct 30 23:58:12.944640 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939345 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 23:58:12.944640 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939348 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 23:58:12.944640 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939351 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 23:58:12.944640 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939357 2574 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 23:58:12.944640 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939360 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 23:58:12.944640 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939363 2574 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 23:58:12.944640 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939366 2574 flags.go:64] FLAG: --logging-format="text" Oct 30 23:58:12.944640 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939369 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 23:58:12.944640 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939372 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 23:58:12.944640 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939375 2574 flags.go:64] FLAG: --manifest-url="" Oct 30 23:58:12.944640 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939378 2574 flags.go:64] FLAG: --manifest-url-header="" Oct 30 23:58:12.944640 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939383 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 23:58:12.944640 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939386 2574 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 23:58:12.944640 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939390 2574 flags.go:64] FLAG: --max-pods="110" Oct 30 23:58:12.945278 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939393 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 23:58:12.945278 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939397 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 23:58:12.945278 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939400 2574 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 23:58:12.945278 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939403 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 23:58:12.945278 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939406 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 23:58:12.945278 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939409 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 23:58:12.945278 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939412 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 23:58:12.945278 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939419 2574 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 23:58:12.945278 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939422 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 23:58:12.945278 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939426 2574 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 23:58:12.945278 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939429 2574 flags.go:64] FLAG: --pod-cidr="" Oct 30 23:58:12.945278 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939432 2574 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 23:58:12.945278 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939435 2574 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 23:58:12.945278 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939439 2574 flags.go:64] FLAG: --pods-per-core="0" Oct 30 23:58:12.945278 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939442 2574 flags.go:64] FLAG: --port="10250" Oct 30 23:58:12.945278 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939445 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 23:58:12.945278 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939448 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-09892ea9ffec88144" Oct 30 23:58:12.945278 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939451 2574 flags.go:64] FLAG: --qos-reserved="" Oct 30 23:58:12.945278 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939454 2574 flags.go:64] FLAG: --read-only-port="10255" Oct 30 23:58:12.945278 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939457 2574 flags.go:64] FLAG: --register-node="true" Oct 30 23:58:12.945278 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939460 2574 flags.go:64] FLAG: --register-with-taints="" Oct 30 23:58:12.945278 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939464 2574 flags.go:64] FLAG: --registry-burst="10" Oct 30 23:58:12.945278 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939466 2574 flags.go:64] FLAG: --registry-qps="5" Oct 30 23:58:12.945278 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939469 2574 flags.go:64] FLAG: --reserved-cpus="" Oct 30 23:58:12.945278 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939472 2574 flags.go:64] FLAG: --reserved-memory="" Oct 30 23:58:12.945278 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939476 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 23:58:12.945872 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939479 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 23:58:12.945872 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939482 2574 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 23:58:12.945872 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939485 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 23:58:12.945872 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939488 2574 flags.go:64] FLAG: --runonce="false" Oct 30 23:58:12.945872 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939491 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 23:58:12.945872 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939494 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 23:58:12.945872 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939497 2574 flags.go:64] FLAG: --seccomp-default="false" Oct 30 23:58:12.945872 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939501 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 23:58:12.945872 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939504 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 23:58:12.945872 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939507 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 23:58:12.945872 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939511 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 23:58:12.945872 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939514 2574 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 23:58:12.945872 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939517 2574 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 23:58:12.945872 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939520 2574 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 23:58:12.945872 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939523 2574 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 23:58:12.945872 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939527 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 23:58:12.945872 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939531 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 23:58:12.945872 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939534 2574 flags.go:64] FLAG: --system-cgroups="" Oct 30 23:58:12.945872 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939538 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 23:58:12.945872 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939544 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 23:58:12.945872 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939547 2574 flags.go:64] FLAG: --tls-cert-file="" Oct 30 23:58:12.945872 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939550 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 23:58:12.945872 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939554 2574 flags.go:64] FLAG: --tls-min-version="" Oct 30 23:58:12.945872 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939556 2574 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 23:58:12.945872 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939559 2574 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 23:58:12.946473 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939562 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 23:58:12.946473 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939565 2574 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 23:58:12.946473 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939568 2574 flags.go:64] FLAG: --v="2" Oct 30 23:58:12.946473 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939573 2574 flags.go:64] FLAG: --version="false" Oct 30 23:58:12.946473 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939577 2574 flags.go:64] FLAG: --vmodule="" Oct 30 23:58:12.946473 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939581 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 23:58:12.946473 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.939585 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 23:58:12.946473 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939718 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:58:12.946473 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939721 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:58:12.946473 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939724 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:58:12.946473 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939727 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:58:12.946473 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939730 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:58:12.946473 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939732 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:58:12.946473 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939735 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:58:12.946473 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939738 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:58:12.946473 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939741 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:58:12.946473 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939743 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:58:12.946473 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939746 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:58:12.946473 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939749 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:58:12.946473 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939751 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:58:12.946473 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939755 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:58:12.946974 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939759 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:58:12.946974 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939764 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:58:12.946974 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939767 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:58:12.946974 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939769 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:58:12.946974 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939774 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:58:12.946974 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939776 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:58:12.946974 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939779 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:58:12.946974 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939782 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:58:12.946974 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939784 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:58:12.946974 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939787 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:58:12.946974 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939789 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:58:12.946974 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939793 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:58:12.946974 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939797 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:58:12.946974 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939800 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:58:12.946974 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939802 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:58:12.946974 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939805 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:58:12.946974 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939808 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:58:12.946974 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939810 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:58:12.946974 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939813 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:58:12.947429 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939816 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:58:12.947429 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939818 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:58:12.947429 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939821 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:58:12.947429 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939824 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:58:12.947429 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939827 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:58:12.947429 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939829 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:58:12.947429 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939832 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:58:12.947429 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939834 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:58:12.947429 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939837 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:58:12.947429 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939840 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:58:12.947429 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939843 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:58:12.947429 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939851 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:58:12.947429 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939855 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:58:12.947429 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939857 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:58:12.947429 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939862 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:58:12.947429 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939865 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:58:12.947429 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939868 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:58:12.947429 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939872 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:58:12.947429 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939874 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:58:12.947894 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939877 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:58:12.947894 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939879 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:58:12.947894 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939882 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:58:12.947894 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939885 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:58:12.947894 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939887 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:58:12.947894 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939890 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:58:12.947894 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939893 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:58:12.947894 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939895 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:58:12.947894 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939898 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:58:12.947894 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939901 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:58:12.947894 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939904 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:58:12.947894 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939906 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:58:12.947894 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939909 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:58:12.947894 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939911 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:58:12.947894 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939914 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:58:12.947894 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939917 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:58:12.947894 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939919 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:58:12.947894 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939922 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:58:12.947894 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939924 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:58:12.947894 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939927 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:58:12.948417 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939929 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:58:12.948417 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939932 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:58:12.948417 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939935 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:58:12.948417 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939938 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:58:12.948417 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939941 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:58:12.948417 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939944 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:58:12.948417 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939946 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:58:12.948417 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939962 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:58:12.948417 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939965 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:58:12.948417 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939968 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:58:12.948417 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939972 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:58:12.948417 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939975 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:58:12.948417 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939977 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:58:12.948417 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939980 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:58:12.948417 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939983 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:58:12.948417 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939986 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:58:12.948417 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939988 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:58:12.948417 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939991 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:58:12.948417 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939994 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:58:12.948417 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939996 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:58:12.948885 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.939999 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:58:12.948885 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940002 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:58:12.948885 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940005 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:58:12.948885 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940007 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:58:12.948885 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940010 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:58:12.948885 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940012 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:58:12.948885 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940015 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:58:12.948885 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940018 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:58:12.948885 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940020 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:58:12.948885 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940023 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:58:12.948885 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940025 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:58:12.948885 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940028 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:58:12.948885 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940030 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:58:12.948885 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940033 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:58:12.948885 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.940061 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:58:12.948885 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940097 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:58:12.949285 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940102 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:58:12.949285 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940106 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:58:12.949285 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940108 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:58:12.949285 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940113 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:58:12.949285 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940116 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:58:12.949285 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940119 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:58:12.949285 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940122 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:58:12.949285 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940126 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:58:12.949285 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940128 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:58:12.949285 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940131 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:58:12.949285 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940134 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:58:12.949285 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940137 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:58:12.949285 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940139 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:58:12.949285 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940142 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:58:12.949285 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940145 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:58:12.949285 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940148 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:58:12.949285 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940150 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:58:12.949285 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940153 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:58:12.949285 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940156 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:58:12.949285 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940158 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:58:12.949743 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940161 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:58:12.949743 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940164 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:58:12.949743 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940167 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:58:12.949743 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940169 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:58:12.949743 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940172 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:58:12.949743 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940174 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:58:12.949743 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940177 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:58:12.949743 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940179 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:58:12.949743 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940182 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:58:12.949743 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940184 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:58:12.949743 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940187 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:58:12.949743 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940189 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:58:12.949743 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940193 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:58:12.949743 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940195 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:58:12.949743 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940198 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:58:12.949743 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940202 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:58:12.949743 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940204 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:58:12.949743 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940207 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:58:12.949743 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940210 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:58:12.949743 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940214 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:58:12.950291 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940216 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:58:12.950291 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940219 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:58:12.950291 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940222 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:58:12.950291 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940224 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:58:12.950291 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940227 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:58:12.950291 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940230 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:58:12.950291 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940233 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:58:12.950291 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940236 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:58:12.950291 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940239 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:58:12.950291 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940242 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:58:12.950291 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940244 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:58:12.950291 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940247 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:58:12.950291 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940249 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:58:12.950291 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940252 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:58:12.950291 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940254 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:58:12.950291 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940258 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:58:12.950291 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940260 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:58:12.950291 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940263 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:58:12.950291 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940267 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:58:12.950729 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940270 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:58:12.950729 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940274 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:58:12.950729 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940277 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:58:12.950729 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940280 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:58:12.950729 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940283 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:58:12.950729 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940286 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:58:12.950729 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940289 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:58:12.950729 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940292 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:58:12.950729 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940296 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:58:12.950729 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940298 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:58:12.950729 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940301 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:58:12.950729 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940304 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:58:12.950729 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940308 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:58:12.950729 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940311 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:58:12.950729 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940314 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:58:12.950729 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940316 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:58:12.950729 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940319 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:58:12.950729 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940321 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:58:12.950729 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940324 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:58:12.951271 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940326 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:58:12.951271 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940329 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:58:12.951271 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940332 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:58:12.951271 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940334 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:58:12.951271 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940337 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:58:12.951271 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940340 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:58:12.951271 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940342 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:58:12.951271 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940345 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:58:12.951271 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940347 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:58:12.951271 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940350 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:58:12.951271 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940353 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:58:12.951271 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940355 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:58:12.951271 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940358 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:58:12.951271 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940360 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:58:12.951271 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940363 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:58:12.951271 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940366 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:58:12.951271 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940368 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:58:12.951271 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940371 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:58:12.951271 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940373 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:58:12.951271 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940376 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:58:12.952053 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940379 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:58:12.952053 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940383 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:58:12.952053 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940387 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:58:12.952053 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940389 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:58:12.952053 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940392 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:58:12.952053 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940395 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:58:12.952053 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.940398 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:58:12.952053 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.947237 2574 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 23:58:12.952053 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.947254 2574 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 23:58:12.952053 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.947271 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 23:58:12.952053 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.947279 2574 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 23:58:12.952053 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947346 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:58:12.952053 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947352 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:58:12.952053 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947355 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:58:12.952053 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947358 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:58:12.952053 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947369 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:58:12.952053 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947372 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:58:12.952053 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947377 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:58:12.952053 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947382 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:58:12.952053 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947386 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:58:12.952599 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947390 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:58:12.952599 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947393 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:58:12.952599 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947396 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:58:12.952599 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947399 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:58:12.952599 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947401 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:58:12.952599 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947404 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:58:12.952599 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947407 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:58:12.952599 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947409 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:58:12.952599 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947412 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:58:12.952599 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947414 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:58:12.952599 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947417 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:58:12.952599 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947419 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:58:12.952599 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947422 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:58:12.952599 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947424 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:58:12.952599 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947427 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:58:12.952599 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947430 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:58:12.952599 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947433 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:58:12.952599 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947435 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:58:12.952599 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947438 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:58:12.953147 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947442 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:58:12.953147 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947452 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:58:12.953147 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947456 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:58:12.953147 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947461 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:58:12.953147 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947465 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:58:12.953147 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947470 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:58:12.953147 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947474 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:58:12.953147 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947478 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:58:12.953147 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947482 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:58:12.953147 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947485 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:58:12.953147 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947489 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:58:12.953147 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947492 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:58:12.953147 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947495 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:58:12.953147 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947498 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:58:12.953147 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947501 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:58:12.953147 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947503 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:58:12.953147 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947506 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:58:12.953147 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947508 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:58:12.953147 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947511 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:58:12.953616 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947514 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:58:12.953616 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947516 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:58:12.953616 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947519 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:58:12.953616 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947522 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:58:12.953616 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947524 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:58:12.953616 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947527 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:58:12.953616 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947529 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:58:12.953616 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947532 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:58:12.953616 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947536 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:58:12.953616 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947540 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:58:12.953616 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947545 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:58:12.953616 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947549 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:58:12.953616 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947553 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:58:12.953616 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947556 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:58:12.953616 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947559 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:58:12.953616 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947571 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:58:12.953616 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947574 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:58:12.953616 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947576 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:58:12.953616 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947579 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:58:12.953616 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947582 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:58:12.954137 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947584 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:58:12.954137 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947586 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:58:12.954137 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947589 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:58:12.954137 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947592 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:58:12.954137 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947594 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:58:12.954137 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947596 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:58:12.954137 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947599 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:58:12.954137 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947601 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:58:12.954137 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947604 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:58:12.954137 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947607 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:58:12.954137 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947609 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:58:12.954137 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947612 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:58:12.954137 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947615 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:58:12.954137 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947619 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:58:12.954137 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947624 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:58:12.954137 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947628 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:58:12.954137 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947632 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:58:12.954137 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947634 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:58:12.954137 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947637 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:58:12.954137 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947639 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:58:12.954589 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947642 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:58:12.954589 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947644 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:58:12.954589 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947647 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:58:12.954589 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947649 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:58:12.954589 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947652 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:58:12.954589 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947655 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:58:12.954589 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947657 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:58:12.954589 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947660 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:58:12.954589 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947669 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:58:12.954589 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947672 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:58:12.954589 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947676 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:58:12.954589 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947680 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:58:12.954589 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947683 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:58:12.954589 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947686 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:58:12.954589 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947690 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:58:12.954589 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947692 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:58:12.954589 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947695 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:58:12.954589 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947699 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:58:12.954589 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947703 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:58:12.955055 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.947733 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:58:12.955055 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947973 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:58:12.955055 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947980 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:58:12.955055 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947982 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:58:12.955055 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947985 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:58:12.955055 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947988 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:58:12.955055 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947991 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:58:12.955055 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947994 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:58:12.955055 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947996 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:58:12.955055 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.947999 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:58:12.955055 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948002 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:58:12.955055 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948005 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:58:12.955055 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948007 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:58:12.955055 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948010 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:58:12.955055 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948012 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:58:12.955055 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948015 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:58:12.955446 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948018 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:58:12.955446 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948021 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:58:12.955446 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948026 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:58:12.955446 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948030 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:58:12.955446 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948036 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:58:12.955446 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948040 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:58:12.955446 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948052 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:58:12.955446 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948055 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:58:12.955446 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948058 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:58:12.955446 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948061 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:58:12.955446 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948063 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:58:12.955446 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948066 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:58:12.955446 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948068 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:58:12.955446 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948071 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:58:12.955446 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948074 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:58:12.955446 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948077 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:58:12.955446 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948079 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:58:12.955446 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948082 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:58:12.955446 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948084 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:58:12.955892 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948087 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:58:12.955892 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948090 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:58:12.955892 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948092 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:58:12.955892 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948095 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:58:12.955892 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948097 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:58:12.955892 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948101 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:58:12.955892 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948105 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:58:12.955892 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948109 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:58:12.955892 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948114 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:58:12.955892 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948118 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:58:12.955892 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948121 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:58:12.955892 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948123 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:58:12.955892 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948126 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:58:12.955892 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948128 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:58:12.955892 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948131 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:58:12.955892 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948134 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:58:12.955892 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948136 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:58:12.955892 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948139 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:58:12.955892 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948141 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:58:12.955892 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948144 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:58:12.956411 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948152 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:58:12.956411 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948155 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:58:12.956411 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948158 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:58:12.956411 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948160 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:58:12.956411 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948163 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:58:12.956411 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948165 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:58:12.956411 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948168 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:58:12.956411 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948170 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:58:12.956411 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948173 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:58:12.956411 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948175 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:58:12.956411 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948178 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:58:12.956411 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948181 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:58:12.956411 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948185 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:58:12.956411 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948189 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:58:12.956411 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948194 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:58:12.956411 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948198 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:58:12.956411 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948201 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:58:12.956411 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948204 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:58:12.956411 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948208 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:58:12.956855 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948212 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:58:12.956855 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948215 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:58:12.956855 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948218 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:58:12.956855 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948221 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:58:12.956855 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948224 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:58:12.956855 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948227 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:58:12.956855 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948230 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:58:12.956855 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948232 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:58:12.956855 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948235 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:58:12.956855 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948237 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:58:12.956855 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948240 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:58:12.956855 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948243 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:58:12.956855 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948245 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:58:12.956855 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948248 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:58:12.956855 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948256 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:58:12.956855 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948259 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:58:12.956855 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948263 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:58:12.956855 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948267 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:58:12.956855 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948271 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:58:12.956855 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948275 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:58:12.957422 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948280 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:58:12.957422 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948283 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:58:12.957422 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948285 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:58:12.957422 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948288 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:58:12.957422 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948291 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:58:12.957422 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948293 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:58:12.957422 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948296 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:58:12.957422 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948299 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:58:12.957422 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948301 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:58:12.957422 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948304 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:58:12.957422 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948306 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:58:12.957422 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948309 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:58:12.957422 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948311 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:58:12.957422 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.948335 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:58:12.957422 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948381 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:58:12.957835 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948386 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:58:12.957835 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948390 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:58:12.957835 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948394 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:58:12.957835 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948397 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:58:12.957835 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948400 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:58:12.957835 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948404 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:58:12.957835 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948406 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:58:12.957835 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948409 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:58:12.957835 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948411 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:58:12.957835 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948414 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:58:12.957835 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948417 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:58:12.957835 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948419 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:58:12.957835 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948430 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:58:12.957835 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948435 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:58:12.957835 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948446 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:58:12.957835 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948450 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:58:12.957835 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948452 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:58:12.957835 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948455 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:58:12.957835 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948457 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:58:12.957835 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948460 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:58:12.958320 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948462 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:58:12.958320 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948465 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:58:12.958320 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948468 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:58:12.958320 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948470 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:58:12.958320 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948473 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:58:12.958320 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948475 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:58:12.958320 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948478 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:58:12.958320 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948481 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:58:12.958320 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948483 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:58:12.958320 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948486 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:58:12.958320 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948488 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:58:12.958320 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948491 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:58:12.958320 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948494 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:58:12.958320 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948496 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:58:12.958320 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948498 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:58:12.958320 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948501 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:58:12.958320 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948508 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:58:12.958320 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948512 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:58:12.958320 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948516 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:58:12.958320 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948520 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:58:12.958818 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948524 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:58:12.958818 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948527 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:58:12.958818 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948530 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:58:12.958818 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948532 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:58:12.958818 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948535 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:58:12.958818 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948543 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:58:12.958818 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948546 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:58:12.958818 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948549 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:58:12.958818 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948551 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:58:12.958818 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948553 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:58:12.958818 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948556 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:58:12.958818 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948558 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:58:12.958818 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948561 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:58:12.958818 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948565 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:58:12.958818 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948569 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:58:12.958818 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948572 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:58:12.958818 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948575 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:58:12.958818 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948577 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:58:12.958818 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948580 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:58:12.959288 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948582 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:58:12.959288 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948585 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:58:12.959288 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948588 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:58:12.959288 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948593 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:58:12.959288 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948597 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:58:12.959288 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948601 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:58:12.959288 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948605 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:58:12.959288 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948608 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:58:12.959288 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948610 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:58:12.959288 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948614 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:58:12.959288 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948617 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:58:12.959288 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948619 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:58:12.959288 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948622 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:58:12.959288 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948624 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:58:12.959288 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948627 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:58:12.959288 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948629 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:58:12.959288 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948632 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:58:12.959288 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948634 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:58:12.959288 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948637 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:58:12.959723 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948647 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:58:12.959723 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948650 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:58:12.959723 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948652 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:58:12.959723 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948655 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:58:12.959723 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948657 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:58:12.959723 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948659 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:58:12.959723 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948662 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:58:12.959723 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948665 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:58:12.959723 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948668 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:58:12.959723 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948672 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:58:12.959723 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948676 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:58:12.959723 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948680 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:58:12.959723 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948685 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:58:12.959723 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948688 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:58:12.959723 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948690 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:58:12.959723 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948693 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:58:12.959723 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948696 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:58:12.959723 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948698 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:58:12.959723 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948701 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:58:12.960369 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948703 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:58:12.960369 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948706 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:58:12.960369 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948708 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:58:12.960369 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948712 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:58:12.960369 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948716 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:58:12.960369 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948718 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:58:12.960369 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948721 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:58:12.960369 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:12.948723 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:58:12.960369 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.948791 2574 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 23:58:12.960369 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.953143 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 23:58:12.960369 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.953998 2574 server.go:1014] "Starting client certificate rotation" Oct 30 23:58:12.960369 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.954090 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 23:58:12.960369 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.954120 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 23:58:12.960369 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.956346 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 23:58:12.960369 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.957699 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 23:58:12.967664 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.967646 2574 log.go:25] "Validated CRI v1 runtime API" Oct 30 23:58:12.970596 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.970581 2574 log.go:25] "Validated CRI v1 image API" Oct 30 23:58:12.971378 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.971356 2574 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 23:58:12.973779 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.973758 2574 fs.go:135] Filesystem UUIDs: map[4c9d44f2-3e4d-4627-a35a-38d3ce7417fd:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d267168e-f8ab-4cee-827b-d48137613b26:/dev/nvme0n1p3] Oct 30 23:58:12.973850 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.973778 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 23:58:12.979567 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.979449 2574 manager.go:217] Machine: {Timestamp:2026-10-30 23:58:12.978283578 +0000 UTC m=+0.295432881 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104764 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24f91baf472d4c7b81fb5c645d05aa SystemUUID:ec24f91b-af47-2d4c-7b81-fb5c645d05aa BootID:c8f29756-d4ce-43e8-89a1-0ba0a1f6f014 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d1:05:7d:bb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d1:05:7d:bb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a2:09:34:f6:e8:dc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 23:58:12.979567 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.979566 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 23:58:12.979694 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.979639 2574 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 23:58:12.982293 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.982274 2574 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 23:58:12.982426 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.982293 2574 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-70.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 23:58:12.982476 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.982428 2574 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 23:58:12.982476 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.982436 2574 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 23:58:12.982476 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.982458 2574 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 23:58:12.982476 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.982470 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 23:58:12.983213 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.983203 2574 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 23:58:12.987465 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.987135 2574 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 23:58:12.987465 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.987400 2574 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 23:58:12.988326 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.988311 2574 kubelet.go:493] "Attempting to sync node with API server" Oct 30 23:58:12.988381 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.988340 2574 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 23:58:12.988381 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.988353 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 23:58:12.988381 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.988361 2574 kubelet.go:405] "Adding apiserver pod source" Oct 30 23:58:12.988381 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.988369 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 23:58:12.988724 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.988709 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 23:58:12.990655 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.990637 2574 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 23:58:12.992242 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.992224 2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 23:58:12.994418 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.994404 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 23:58:12.995229 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.995215 2574 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 23:58:12.995287 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.995244 2574 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 23:58:12.995394 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.995383 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 23:58:12.995427 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.995404 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 23:58:12.995427 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.995412 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 23:58:12.995427 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.995421 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 23:58:12.995506 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.995429 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 23:58:12.995506 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.995440 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 23:58:12.995990 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.995980 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 23:58:12.996027 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.995990 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 23:58:12.996027 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.995998 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 23:58:12.996027 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.996004 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 23:58:12.996027 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.996013 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 23:58:12.996027 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.996022 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 23:58:12.996587 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.996578 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 23:58:12.996625 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:12.996590 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 23:58:13.000243 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.000229 2574 server.go:1263] "Started kubelet" Oct 30 23:58:13.000311 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.000287 2574 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 23:58:13.000514 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.000410 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 23:58:13.000597 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.000587 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 23:58:13.000919 ip-10-0-129-70 systemd[1]: Started Kubernetes Kubelet. Oct 30 23:58:13.003651 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.003020 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-70.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 23:58:13.003732 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.003569 2574 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 23:58:13.005593 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.005435 2574 server.go:317] "Adding debug handlers to kubelet server" Oct 30 23:58:13.006118 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:13.006076 2574 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 23:58:13.007909 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:13.007888 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-70.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 23:58:13.007909 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:13.007893 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 23:58:13.008131 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:13.007194 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-70.us-east-2.compute.internal.18e3741100cbb195 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-70.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-70.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 23:58:13.000204693 +0000 UTC m=+0.317354021,LastTimestamp:2026-10-30 23:58:13.000204693 +0000 UTC m=+0.317354021,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-70.us-east-2.compute.internal,}" Oct 30 23:58:13.008875 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.008859 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 23:58:13.009216 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.009198 2574 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 23:58:13.009336 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:13.009316 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-70.us-east-2.compute.internal\" not found" Oct 30 23:58:13.009891 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.009873 2574 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 23:58:13.009983 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.009899 2574 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 23:58:13.010023 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.009984 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 23:58:13.010065 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.010026 2574 reconstruct.go:97] "Volume reconstruction finished" Oct 30 23:58:13.010065 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.010033 2574 reconciler.go:29] "Reconciler: start to sync state" Oct 30 23:58:13.010208 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.010189 2574 factory.go:55] Registering systemd factory Oct 30 23:58:13.010208 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.010203 2574 factory.go:223] Registration of the systemd container factory successfully Oct 30 23:58:13.010438 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.010423 2574 factory.go:153] Registering CRI-O factory Oct 30 23:58:13.010483 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.010443 2574 factory.go:223] Registration of the crio container factory successfully Oct 30 23:58:13.010527 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.010503 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 23:58:13.010527 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.010524 2574 factory.go:103] Registering Raw factory Oct 30 23:58:13.010602 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.010537 2574 manager.go:1199] Started watching for new ooms in manager Oct 30 23:58:13.011461 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.011445 2574 manager.go:319] Starting recovery of all containers Oct 30 23:58:13.011542 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:13.011489 2574 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-70.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 23:58:13.015173 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.015150 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7x647" Oct 30 23:58:13.018578 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:13.018533 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 23:58:13.023690 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.023671 2574 manager.go:324] Recovery completed Oct 30 23:58:13.026144 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.026131 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7x647" Oct 30 23:58:13.029259 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.029248 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:58:13.030740 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.030723 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-70.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:58:13.030828 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.030755 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-70.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:58:13.030828 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.030769 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-70.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:58:13.031329 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.031311 2574 cpu_manager.go:226] "Starting" policy="none" Oct 30 23:58:13.031329 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.031327 2574 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 23:58:13.031447 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.031316 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:58:13.031447 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.031343 2574 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 23:58:13.032035 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.032019 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-70.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:58:13.032098 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.032049 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-70.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:58:13.032098 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.032059 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-70.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:58:13.032765 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:13.032707 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-70.us-east-2.compute.internal.18e37411029d9e6e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-70.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-70.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-70.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 23:58:13.030739566 +0000 UTC m=+0.347888875,LastTimestamp:2026-10-30 23:58:13.030739566 +0000 UTC m=+0.347888875,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-70.us-east-2.compute.internal,}" Oct 30 23:58:13.033313 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.033302 2574 policy_none.go:50] "Start" Oct 30 23:58:13.033364 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.033317 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 23:58:13.033364 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.033326 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 23:58:13.035329 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.035315 2574 policy_none.go:44] "Start" Oct 30 23:58:13.058616 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.058590 2574 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 23:58:13.061304 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.061287 2574 manager.go:341] "Starting Device Plugin manager" Oct 30 23:58:13.061778 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:13.061761 2574 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 23:58:13.061832 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.061789 2574 server.go:87] "Starting device plugin registration server" Oct 30 23:58:13.061995 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.061984 2574 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 23:58:13.062042 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.061996 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 23:58:13.062112 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.062097 2574 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 23:58:13.062204 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.062193 2574 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 23:58:13.062246 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.062204 2574 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 23:58:13.062691 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:13.062672 2574 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 23:58:13.062763 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:13.062716 2574 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-70.us-east-2.compute.internal\" not found" Oct 30 23:58:13.152261 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.152106 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 23:58:13.153403 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.153387 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 23:58:13.153476 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.153411 2574 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 23:58:13.153476 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.153433 2574 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 23:58:13.153573 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:13.153471 2574 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 23:58:13.157141 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.157123 2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:58:13.162216 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.162201 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:58:13.162906 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.162892 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-70.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:58:13.162975 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.162923 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-70.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:58:13.162975 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.162935 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-70.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:58:13.163047 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.162979 2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:58:13.171454 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.171440 2574 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:58:13.171508 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:13.171460 2574 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-70.us-east-2.compute.internal\": node \"ip-10-0-129-70.us-east-2.compute.internal\" not found" Oct 30 23:58:13.185774 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:13.185736 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-70.us-east-2.compute.internal\" not found" Oct 30 23:58:13.253714 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.253684 2574 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-70.us-east-2.compute.internal"] Oct 30 23:58:13.253808 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.253761 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:58:13.254561 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.254536 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-70.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:58:13.254561 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.254560 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-70.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:58:13.254676 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.254578 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-70.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:58:13.256611 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.256599 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:58:13.256739 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.256724 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:58:13.256785 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.256755 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:58:13.257245 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.257227 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-70.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:58:13.257245 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.257238 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-70.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:58:13.257374 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.257255 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-70.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:58:13.257374 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.257258 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-70.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:58:13.257374 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.257270 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-70.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:58:13.257374 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.257274 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-70.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:58:13.259269 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.259256 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:58:13.259335 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.259277 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:58:13.259986 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.259973 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-70.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:58:13.260038 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.260002 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-70.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:58:13.260038 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.260034 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-70.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:58:13.279218 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:13.279202 2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-70.us-east-2.compute.internal\" not found" node="ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:58:13.281148 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:13.281133 2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-70.us-east-2.compute.internal\" not found" node="ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:58:13.286584 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:13.286570 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-70.us-east-2.compute.internal\" not found" Oct 30 23:58:13.387268 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:13.387250 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-70.us-east-2.compute.internal\" not found" Oct 30 23:58:13.411548 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.411526 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c4b01b095f68d38f7d23560b244b5e01-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal\" (UID: \"c4b01b095f68d38f7d23560b244b5e01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:58:13.411602 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.411553 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4b01b095f68d38f7d23560b244b5e01-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal\" (UID: \"c4b01b095f68d38f7d23560b244b5e01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:58:13.411602 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.411570 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2e9a3bb1bd7fbf1e3c59a1731918cc24-config\") pod \"kube-apiserver-proxy-ip-10-0-129-70.us-east-2.compute.internal\" (UID: \"2e9a3bb1bd7fbf1e3c59a1731918cc24\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:58:13.487793 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:13.487748 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-70.us-east-2.compute.internal\" not found" Oct 30 23:58:13.512263 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.512241 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c4b01b095f68d38f7d23560b244b5e01-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal\" (UID: \"c4b01b095f68d38f7d23560b244b5e01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:58:13.512338 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.512281 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4b01b095f68d38f7d23560b244b5e01-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal\" (UID: \"c4b01b095f68d38f7d23560b244b5e01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:58:13.512338 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.512297 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2e9a3bb1bd7fbf1e3c59a1731918cc24-config\") pod \"kube-apiserver-proxy-ip-10-0-129-70.us-east-2.compute.internal\" (UID: \"2e9a3bb1bd7fbf1e3c59a1731918cc24\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:58:13.512338 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.512325 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4b01b095f68d38f7d23560b244b5e01-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal\" (UID: \"c4b01b095f68d38f7d23560b244b5e01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:58:13.512432 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.512334 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c4b01b095f68d38f7d23560b244b5e01-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal\" (UID: \"c4b01b095f68d38f7d23560b244b5e01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:58:13.512432 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.512381 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2e9a3bb1bd7fbf1e3c59a1731918cc24-config\") pod \"kube-apiserver-proxy-ip-10-0-129-70.us-east-2.compute.internal\" (UID: \"2e9a3bb1bd7fbf1e3c59a1731918cc24\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:58:13.581349 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.581331 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:58:13.583494 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.583478 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:58:13.587821 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:13.587803 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-70.us-east-2.compute.internal\" not found" Oct 30 23:58:13.688394 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:13.688371 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-70.us-east-2.compute.internal\" not found" Oct 30 23:58:13.788844 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:13.788795 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-70.us-east-2.compute.internal\" not found" Oct 30 23:58:13.870402 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.870383 2574 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:58:13.909732 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.909715 2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:58:13.921891 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.921876 2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:58:13.923350 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.923338 2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:58:13.932318 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.932303 2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:58:13.953395 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.953375 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 23:58:13.953517 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.953497 2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 23:58:13.953555 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.953536 2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 23:58:13.989197 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:13.989181 2574 apiserver.go:52] "Watching apiserver" Oct 30 23:58:14.004637 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.004612 2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:58:14.007995 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.007979 2574 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 23:58:14.008267 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.008250 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-x962k","openshift-network-operator/iptables-alerter-zlgls","kube-system/kube-apiserver-proxy-ip-10-0-129-70.us-east-2.compute.internal","openshift-dns/node-resolver-6hq7b","openshift-image-registry/node-ca-vnk6j","openshift-multus/multus-9j82q","kube-system/global-pull-secret-syncer-6l8ht","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w","openshift-network-diagnostics/network-check-target-bd4bx","openshift-ovn-kubernetes/ovnkube-node-7zxsz","openshift-cluster-node-tuning-operator/tuned-f64c8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-tc87d","openshift-multus/network-metrics-daemon-cjqfw"] Oct 30 23:58:14.009785 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.009763 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 23:58:14.010622 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.010600 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6hq7b" Oct 30 23:58:14.012552 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.012533 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zlgls" Oct 30 23:58:14.014645 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.014629 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:58:14.014986 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.014966 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d7815306-7a99-4fc5-8226-b34916ceaaa2-hosts-file\") pod \"node-resolver-6hq7b\" (UID: \"d7815306-7a99-4fc5-8226-b34916ceaaa2\") " pod="openshift-dns/node-resolver-6hq7b" Oct 30 23:58:14.015051 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.014994 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7815306-7a99-4fc5-8226-b34916ceaaa2-tmp-dir\") pod \"node-resolver-6hq7b\" (UID: \"d7815306-7a99-4fc5-8226-b34916ceaaa2\") " pod="openshift-dns/node-resolver-6hq7b" Oct 30 23:58:14.015051 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.015010 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kjxh\" (UniqueName: \"kubernetes.io/projected/d7815306-7a99-4fc5-8226-b34916ceaaa2-kube-api-access-2kjxh\") pod \"node-resolver-6hq7b\" (UID: \"d7815306-7a99-4fc5-8226-b34916ceaaa2\") " pod="openshift-dns/node-resolver-6hq7b" Oct 30 23:58:14.016681 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.016635 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:58:14.016801 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:14.016710 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6l8ht" podUID="99b4d92d-2003-4708-99da-a5f12cbb10e2" Oct 30 23:58:14.018609 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.018591 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 23:58:14.018832 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.018814 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vnk6j" Oct 30 23:58:14.019016 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.018845 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 23:58:14.019244 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.019221 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:58:14.019335 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.019271 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 23:58:14.019335 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.019277 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 23:58:14.019335 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.019298 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 23:58:14.019335 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.019309 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 23:58:14.019335 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.019322 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 23:58:14.019335 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.019297 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 23:58:14.019530 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.019308 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 23:58:14.019818 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.019805 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pxpmk\"" Oct 30 23:58:14.020979 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.020947 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.021308 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.021293 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-72z8g\"" Oct 30 23:58:14.021389 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.021350 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rqcpl\"" Oct 30 23:58:14.023107 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.023089 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x962k" Oct 30 23:58:14.025239 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.025221 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:58:14.027327 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.027310 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:58:14.027416 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:14.027356 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4bx" podUID="50c15c32-e5cd-426c-9e7d-c123f8480653" Oct 30 23:58:14.029911 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.029895 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.032019 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.031985 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 23:53:13 +0000 UTC" deadline="2028-07-25 05:35:36.75432576 +0000 UTC" Oct 30 23:58:14.032019 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.032018 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15197h37m22.722310357s" Oct 30 23:58:14.032275 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.032261 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.034430 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.034414 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:58:14.034498 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:14.034460 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjqfw" podUID="0adc098d-57a4-490e-89b7-b1b271a1c087" Oct 30 23:58:14.039989 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.039929 2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 23:58:14.074736 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.074712 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-667cr" Oct 30 23:58:14.086829 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.086811 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-667cr" Oct 30 23:58:14.111056 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.111042 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 23:58:14.115361 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.115342 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/25e99515-222f-4346-9fe5-5b7b3b213ee8-ovnkube-script-lib\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.115479 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.115369 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8089437-f546-441e-b121-6e79878d2608-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tc87d\" (UID: \"d8089437-f546-441e-b121-6e79878d2608\") " pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:58:14.115479 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.115386 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/63e868ff-3f36-4542-9e1d-9688ce8c37f1-device-dir\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:58:14.115479 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.115445 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djj7v\" (UniqueName: \"kubernetes.io/projected/0adc098d-57a4-490e-89b7-b1b271a1c087-kube-api-access-djj7v\") pod \"network-metrics-daemon-cjqfw\" (UID: \"0adc098d-57a4-490e-89b7-b1b271a1c087\") " pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:58:14.115579 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.115502 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8tds\" (UniqueName: \"kubernetes.io/projected/1f653a44-907a-417e-a1da-9b1bb0ee7283-kube-api-access-r8tds\") pod \"iptables-alerter-zlgls\" (UID: \"1f653a44-907a-417e-a1da-9b1bb0ee7283\") " pod="openshift-network-operator/iptables-alerter-zlgls" Oct 30 23:58:14.115579 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.115522 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/18acd0a8-988e-4168-84e7-0b25dc7b32d4-host\") pod \"node-ca-vnk6j\" (UID: \"18acd0a8-988e-4168-84e7-0b25dc7b32d4\") " pod="openshift-image-registry/node-ca-vnk6j" Oct 30 23:58:14.115579 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.115538 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-host-run-netns\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.115688 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.115575 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-run-netns\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.115688 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.115605 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-host-var-lib-cni-multus\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.115688 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.115625 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a0be3213-788c-45b4-8fbd-37e744a14f17-multus-daemon-config\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.115688 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.115650 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-etc-kubernetes\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.115826 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.115698 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kjxh\" (UniqueName: \"kubernetes.io/projected/d7815306-7a99-4fc5-8226-b34916ceaaa2-kube-api-access-2kjxh\") pod \"node-resolver-6hq7b\" (UID: \"d7815306-7a99-4fc5-8226-b34916ceaaa2\") " pod="openshift-dns/node-resolver-6hq7b" Oct 30 23:58:14.115826 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.115736 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8089437-f546-441e-b121-6e79878d2608-system-cni-dir\") pod \"multus-additional-cni-plugins-tc87d\" (UID: \"d8089437-f546-441e-b121-6e79878d2608\") " pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:58:14.115826 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.115774 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/63e868ff-3f36-4542-9e1d-9688ce8c37f1-socket-dir\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:58:14.115826 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.115805 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-sysctl-conf\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.116015 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.115898 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-lib-modules\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.116015 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.115918 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-system-cni-dir\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.116015 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.115983 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-host-var-lib-cni-bin\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.116186 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.116049 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-multus-conf-dir\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.116186 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.116085 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-run-ovn\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.116186 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.116117 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/99b4d92d-2003-4708-99da-a5f12cbb10e2-kubelet-config\") pod \"global-pull-secret-syncer-6l8ht\" (UID: \"99b4d92d-2003-4708-99da-a5f12cbb10e2\") " pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:58:14.116186 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.116163 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/63e868ff-3f36-4542-9e1d-9688ce8c37f1-registration-dir\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:58:14.116386 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.116192 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-modprobe-d\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.116386 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.116228 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-systemd\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.116386 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.116265 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-kubelet\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.116386 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.116291 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d8089437-f546-441e-b121-6e79878d2608-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tc87d\" (UID: \"d8089437-f546-441e-b121-6e79878d2608\") " pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:58:14.116386 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.116333 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcvd6\" (UniqueName: \"kubernetes.io/projected/5405bfc2-cfc0-4ea6-80bc-911e117f1843-kube-api-access-mcvd6\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.116605 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.116385 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1f653a44-907a-417e-a1da-9b1bb0ee7283-iptables-alerter-script\") pod \"iptables-alerter-zlgls\" (UID: \"1f653a44-907a-417e-a1da-9b1bb0ee7283\") " pod="openshift-network-operator/iptables-alerter-zlgls" Oct 30 23:58:14.116605 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.116422 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmcgf\" (UniqueName: \"kubernetes.io/projected/18acd0a8-988e-4168-84e7-0b25dc7b32d4-kube-api-access-pmcgf\") pod \"node-ca-vnk6j\" (UID: \"18acd0a8-988e-4168-84e7-0b25dc7b32d4\") " pod="openshift-image-registry/node-ca-vnk6j" Oct 30 23:58:14.116605 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.116448 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-multus-socket-dir-parent\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.116605 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.116474 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpxpg\" (UniqueName: \"kubernetes.io/projected/25e99515-222f-4346-9fe5-5b7b3b213ee8-kube-api-access-dpxpg\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.116605 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.116502 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d8089437-f546-441e-b121-6e79878d2608-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tc87d\" (UID: \"d8089437-f546-441e-b121-6e79878d2608\") " pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:58:14.116605 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.116546 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/63e868ff-3f36-4542-9e1d-9688ce8c37f1-sys-fs\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:58:14.116605 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.116578 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd4j2\" (UniqueName: \"kubernetes.io/projected/50c15c32-e5cd-426c-9e7d-c123f8480653-kube-api-access-gd4j2\") pod \"network-check-target-bd4bx\" (UID: \"50c15c32-e5cd-426c-9e7d-c123f8480653\") " pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:58:14.116605 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.116602 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-kubernetes\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.116920 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.116643 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-cnibin\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.116920 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.116711 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/99b4d92d-2003-4708-99da-a5f12cbb10e2-dbus\") pod \"global-pull-secret-syncer-6l8ht\" (UID: \"99b4d92d-2003-4708-99da-a5f12cbb10e2\") " pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:58:14.116920 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.116741 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1f653a44-907a-417e-a1da-9b1bb0ee7283-host-slash\") pod \"iptables-alerter-zlgls\" (UID: \"1f653a44-907a-417e-a1da-9b1bb0ee7283\") " pod="openshift-network-operator/iptables-alerter-zlgls" Oct 30 23:58:14.116920 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.116770 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7815306-7a99-4fc5-8226-b34916ceaaa2-tmp-dir\") pod \"node-resolver-6hq7b\" (UID: \"d7815306-7a99-4fc5-8226-b34916ceaaa2\") " pod="openshift-dns/node-resolver-6hq7b" Oct 30 23:58:14.116920 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.116797 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-etc-openvswitch\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.116920 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.116828 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-hostroot\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.116920 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.116850 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-systemd-units\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.116920 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.116873 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-var-lib-openvswitch\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.117252 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.117109 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7815306-7a99-4fc5-8226-b34916ceaaa2-tmp-dir\") pod \"node-resolver-6hq7b\" (UID: \"d7815306-7a99-4fc5-8226-b34916ceaaa2\") " pod="openshift-dns/node-resolver-6hq7b" Oct 30 23:58:14.117252 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.117189 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0adc098d-57a4-490e-89b7-b1b271a1c087-metrics-certs\") pod \"network-metrics-daemon-cjqfw\" (UID: \"0adc098d-57a4-490e-89b7-b1b271a1c087\") " pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:58:14.117423 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.117262 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-host-run-multus-certs\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.117423 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.117290 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/25e99515-222f-4346-9fe5-5b7b3b213ee8-ovnkube-config\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.117423 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.117339 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/25e99515-222f-4346-9fe5-5b7b3b213ee8-ovn-node-metrics-cert\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.117423 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.117362 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/18acd0a8-988e-4168-84e7-0b25dc7b32d4-serviceca\") pod \"node-ca-vnk6j\" (UID: \"18acd0a8-988e-4168-84e7-0b25dc7b32d4\") " pod="openshift-image-registry/node-ca-vnk6j" Oct 30 23:58:14.117423 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.117378 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-host-run-k8s-cni-cncf-io\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.117423 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.117411 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-host-var-lib-kubelet\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.117668 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.117440 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-slash\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.117668 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.117495 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-run-ovn-kubernetes\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.117668 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.117517 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/63e868ff-3f36-4542-9e1d-9688ce8c37f1-etc-selinux\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:58:14.117668 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.117535 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d7815306-7a99-4fc5-8226-b34916ceaaa2-hosts-file\") pod \"node-resolver-6hq7b\" (UID: \"d7815306-7a99-4fc5-8226-b34916ceaaa2\") " pod="openshift-dns/node-resolver-6hq7b" Oct 30 23:58:14.117668 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.117571 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-sysconfig\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.117668 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.117602 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-run\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.117668 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.117622 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-host\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.117668 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.117620 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d7815306-7a99-4fc5-8226-b34916ceaaa2-hosts-file\") pod \"node-resolver-6hq7b\" (UID: \"d7815306-7a99-4fc5-8226-b34916ceaaa2\") " pod="openshift-dns/node-resolver-6hq7b" Oct 30 23:58:14.117668 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.117656 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5405bfc2-cfc0-4ea6-80bc-911e117f1843-tmp\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.117971 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.117683 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a0be3213-788c-45b4-8fbd-37e744a14f17-cni-binary-copy\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.117971 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.117716 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-run-systemd\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.117971 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.117746 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-cni-netd\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.117971 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.117798 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.117971 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.117816 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/25e99515-222f-4346-9fe5-5b7b3b213ee8-env-overrides\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.117971 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.117830 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8089437-f546-441e-b121-6e79878d2608-cni-binary-copy\") pod \"multus-additional-cni-plugins-tc87d\" (UID: \"d8089437-f546-441e-b121-6e79878d2608\") " pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:58:14.117971 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.117844 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8wrl\" (UniqueName: \"kubernetes.io/projected/d8089437-f546-441e-b121-6e79878d2608-kube-api-access-d8wrl\") pod \"multus-additional-cni-plugins-tc87d\" (UID: \"d8089437-f546-441e-b121-6e79878d2608\") " pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:58:14.117971 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.117860 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/409b7296-1d35-44e9-be1a-3459f0dbaad9-agent-certs\") pod \"konnectivity-agent-x962k\" (UID: \"409b7296-1d35-44e9-be1a-3459f0dbaad9\") " pod="kube-system/konnectivity-agent-x962k" Oct 30 23:58:14.117971 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.117911 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-node-log\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.117971 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.117934 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-cni-bin\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.117971 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.117965 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl2s5\" (UniqueName: \"kubernetes.io/projected/63e868ff-3f36-4542-9e1d-9688ce8c37f1-kube-api-access-fl2s5\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:58:14.118445 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.118017 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-tuned\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.118445 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.118077 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-multus-cni-dir\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.118445 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.118109 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlqqp\" (UniqueName: \"kubernetes.io/projected/a0be3213-788c-45b4-8fbd-37e744a14f17-kube-api-access-hlqqp\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.118445 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.118165 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-run-openvswitch\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.118445 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.118200 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-log-socket\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.118445 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.118240 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8089437-f546-441e-b121-6e79878d2608-cnibin\") pod \"multus-additional-cni-plugins-tc87d\" (UID: \"d8089437-f546-441e-b121-6e79878d2608\") " pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:58:14.118445 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.118316 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99b4d92d-2003-4708-99da-a5f12cbb10e2-original-pull-secret\") pod \"global-pull-secret-syncer-6l8ht\" (UID: \"99b4d92d-2003-4708-99da-a5f12cbb10e2\") " pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:58:14.118445 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.118338 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/63e868ff-3f36-4542-9e1d-9688ce8c37f1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:58:14.118445 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.118353 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-sysctl-d\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.118445 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.118431 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 23:58:14.118835 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.118435 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-sys\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.118835 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.118462 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 23:58:14.118835 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.118455 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dtg7z\"" Oct 30 23:58:14.118835 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.118505 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-var-lib-kubelet\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.118835 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.118554 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/409b7296-1d35-44e9-be1a-3459f0dbaad9-konnectivity-ca\") pod \"konnectivity-agent-x962k\" (UID: \"409b7296-1d35-44e9-be1a-3459f0dbaad9\") " pod="kube-system/konnectivity-agent-x962k" Oct 30 23:58:14.118835 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.118607 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 23:58:14.118835 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.118618 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hrv6t\"" Oct 30 23:58:14.118835 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.118629 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 23:58:14.118835 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.118648 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 23:58:14.118835 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.118681 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 23:58:14.118835 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.118701 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-r8kbq\"" Oct 30 23:58:14.119211 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.119198 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 23:58:14.119264 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.119252 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 23:58:14.119264 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.119255 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 23:58:14.119264 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.119261 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 23:58:14.119392 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.119278 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 23:58:14.119392 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.119295 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 23:58:14.119392 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.119305 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 23:58:14.119392 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.119311 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:58:14.119392 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.119242 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 23:58:14.119392 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.119343 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 23:58:14.119392 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.119345 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-g645g\"" Oct 30 23:58:14.119392 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.119391 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-6ff8s\"" Oct 30 23:58:14.119650 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.119389 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9ccbg\"" Oct 30 23:58:14.120139 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.120124 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 23:58:14.125786 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.125768 2574 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 23:58:14.128728 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.128708 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kjxh\" (UniqueName: \"kubernetes.io/projected/d7815306-7a99-4fc5-8226-b34916ceaaa2-kube-api-access-2kjxh\") pod \"node-resolver-6hq7b\" (UID: \"d7815306-7a99-4fc5-8226-b34916ceaaa2\") " pod="openshift-dns/node-resolver-6hq7b" Oct 30 23:58:14.219185 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219157 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8tds\" (UniqueName: \"kubernetes.io/projected/1f653a44-907a-417e-a1da-9b1bb0ee7283-kube-api-access-r8tds\") pod \"iptables-alerter-zlgls\" (UID: \"1f653a44-907a-417e-a1da-9b1bb0ee7283\") " pod="openshift-network-operator/iptables-alerter-zlgls" Oct 30 23:58:14.219185 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219186 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/18acd0a8-988e-4168-84e7-0b25dc7b32d4-host\") pod \"node-ca-vnk6j\" (UID: \"18acd0a8-988e-4168-84e7-0b25dc7b32d4\") " pod="openshift-image-registry/node-ca-vnk6j" Oct 30 23:58:14.219364 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219202 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-host-run-netns\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.219364 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219218 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-run-netns\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.219364 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219234 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-host-var-lib-cni-multus\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.219364 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219257 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a0be3213-788c-45b4-8fbd-37e744a14f17-multus-daemon-config\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.219364 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219291 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-run-netns\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.219364 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219316 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-host-var-lib-cni-multus\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.219364 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219326 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/18acd0a8-988e-4168-84e7-0b25dc7b32d4-host\") pod \"node-ca-vnk6j\" (UID: \"18acd0a8-988e-4168-84e7-0b25dc7b32d4\") " pod="openshift-image-registry/node-ca-vnk6j" Oct 30 23:58:14.219364 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219294 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-host-run-netns\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.219661 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219408 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-etc-kubernetes\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.219661 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219444 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8089437-f546-441e-b121-6e79878d2608-system-cni-dir\") pod \"multus-additional-cni-plugins-tc87d\" (UID: \"d8089437-f546-441e-b121-6e79878d2608\") " pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:58:14.219661 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219471 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/63e868ff-3f36-4542-9e1d-9688ce8c37f1-socket-dir\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:58:14.219661 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219497 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-sysctl-conf\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.219661 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219508 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-etc-kubernetes\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.219661 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219521 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-lib-modules\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.219661 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219545 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-system-cni-dir\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.219661 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219545 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8089437-f546-441e-b121-6e79878d2608-system-cni-dir\") pod \"multus-additional-cni-plugins-tc87d\" (UID: \"d8089437-f546-441e-b121-6e79878d2608\") " pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:58:14.219661 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219570 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-host-var-lib-cni-bin\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.219661 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219594 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-multus-conf-dir\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.219661 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219631 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/63e868ff-3f36-4542-9e1d-9688ce8c37f1-socket-dir\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:58:14.219661 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219619 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-run-ovn\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.219661 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219662 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-lib-modules\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.220108 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219676 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-run-ovn\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.220108 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219678 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/99b4d92d-2003-4708-99da-a5f12cbb10e2-kubelet-config\") pod \"global-pull-secret-syncer-6l8ht\" (UID: \"99b4d92d-2003-4708-99da-a5f12cbb10e2\") " pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:58:14.220108 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219709 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a0be3213-788c-45b4-8fbd-37e744a14f17-multus-daemon-config\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.220108 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219716 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/63e868ff-3f36-4542-9e1d-9688ce8c37f1-registration-dir\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:58:14.220108 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219718 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/99b4d92d-2003-4708-99da-a5f12cbb10e2-kubelet-config\") pod \"global-pull-secret-syncer-6l8ht\" (UID: \"99b4d92d-2003-4708-99da-a5f12cbb10e2\") " pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:58:14.220108 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219755 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-host-var-lib-cni-bin\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.220108 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219772 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-multus-conf-dir\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.220108 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219778 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/63e868ff-3f36-4542-9e1d-9688ce8c37f1-registration-dir\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:58:14.220108 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219782 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-modprobe-d\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.220108 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219772 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-sysctl-conf\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.220108 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219812 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-systemd\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.220108 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219823 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-system-cni-dir\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.220108 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219835 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-kubelet\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.220108 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219861 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d8089437-f546-441e-b121-6e79878d2608-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tc87d\" (UID: \"d8089437-f546-441e-b121-6e79878d2608\") " pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:58:14.220108 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219869 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-modprobe-d\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.220108 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219877 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-kubelet\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.220108 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219887 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-systemd\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.220935 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219888 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcvd6\" (UniqueName: \"kubernetes.io/projected/5405bfc2-cfc0-4ea6-80bc-911e117f1843-kube-api-access-mcvd6\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.220935 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219933 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1f653a44-907a-417e-a1da-9b1bb0ee7283-iptables-alerter-script\") pod \"iptables-alerter-zlgls\" (UID: \"1f653a44-907a-417e-a1da-9b1bb0ee7283\") " pod="openshift-network-operator/iptables-alerter-zlgls" Oct 30 23:58:14.220935 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219960 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmcgf\" (UniqueName: \"kubernetes.io/projected/18acd0a8-988e-4168-84e7-0b25dc7b32d4-kube-api-access-pmcgf\") pod \"node-ca-vnk6j\" (UID: \"18acd0a8-988e-4168-84e7-0b25dc7b32d4\") " pod="openshift-image-registry/node-ca-vnk6j" Oct 30 23:58:14.220935 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.219986 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-multus-socket-dir-parent\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.220935 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220023 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpxpg\" (UniqueName: \"kubernetes.io/projected/25e99515-222f-4346-9fe5-5b7b3b213ee8-kube-api-access-dpxpg\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.220935 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220039 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d8089437-f546-441e-b121-6e79878d2608-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tc87d\" (UID: \"d8089437-f546-441e-b121-6e79878d2608\") " pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:58:14.220935 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220068 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/63e868ff-3f36-4542-9e1d-9688ce8c37f1-sys-fs\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:58:14.220935 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220094 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd4j2\" (UniqueName: \"kubernetes.io/projected/50c15c32-e5cd-426c-9e7d-c123f8480653-kube-api-access-gd4j2\") pod \"network-check-target-bd4bx\" (UID: \"50c15c32-e5cd-426c-9e7d-c123f8480653\") " pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:58:14.220935 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220128 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-kubernetes\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.220935 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220104 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-multus-socket-dir-parent\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.220935 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220159 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-cnibin\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.220935 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220201 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/99b4d92d-2003-4708-99da-a5f12cbb10e2-dbus\") pod \"global-pull-secret-syncer-6l8ht\" (UID: \"99b4d92d-2003-4708-99da-a5f12cbb10e2\") " pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:58:14.220935 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220227 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1f653a44-907a-417e-a1da-9b1bb0ee7283-host-slash\") pod \"iptables-alerter-zlgls\" (UID: \"1f653a44-907a-417e-a1da-9b1bb0ee7283\") " pod="openshift-network-operator/iptables-alerter-zlgls" Oct 30 23:58:14.220935 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220270 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-etc-openvswitch\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.220935 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220269 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/63e868ff-3f36-4542-9e1d-9688ce8c37f1-sys-fs\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:58:14.220935 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220294 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-hostroot\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.220935 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220318 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-systemd-units\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.221807 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220348 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-var-lib-openvswitch\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.221807 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220374 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0adc098d-57a4-490e-89b7-b1b271a1c087-metrics-certs\") pod \"network-metrics-daemon-cjqfw\" (UID: \"0adc098d-57a4-490e-89b7-b1b271a1c087\") " pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:58:14.221807 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220396 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/99b4d92d-2003-4708-99da-a5f12cbb10e2-dbus\") pod \"global-pull-secret-syncer-6l8ht\" (UID: \"99b4d92d-2003-4708-99da-a5f12cbb10e2\") " pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:58:14.221807 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220407 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-host-run-multus-certs\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.221807 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220435 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/25e99515-222f-4346-9fe5-5b7b3b213ee8-ovnkube-config\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.221807 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220450 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-kubernetes\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.221807 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220467 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1f653a44-907a-417e-a1da-9b1bb0ee7283-iptables-alerter-script\") pod \"iptables-alerter-zlgls\" (UID: \"1f653a44-907a-417e-a1da-9b1bb0ee7283\") " pod="openshift-network-operator/iptables-alerter-zlgls" Oct 30 23:58:14.221807 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220474 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d8089437-f546-441e-b121-6e79878d2608-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tc87d\" (UID: \"d8089437-f546-441e-b121-6e79878d2608\") " pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:58:14.221807 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220501 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-cnibin\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.221807 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220509 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/25e99515-222f-4346-9fe5-5b7b3b213ee8-ovn-node-metrics-cert\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.221807 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220521 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1f653a44-907a-417e-a1da-9b1bb0ee7283-host-slash\") pod \"iptables-alerter-zlgls\" (UID: \"1f653a44-907a-417e-a1da-9b1bb0ee7283\") " pod="openshift-network-operator/iptables-alerter-zlgls" Oct 30 23:58:14.221807 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220536 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/18acd0a8-988e-4168-84e7-0b25dc7b32d4-serviceca\") pod \"node-ca-vnk6j\" (UID: \"18acd0a8-988e-4168-84e7-0b25dc7b32d4\") " pod="openshift-image-registry/node-ca-vnk6j" Oct 30 23:58:14.221807 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220546 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-etc-openvswitch\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.221807 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220548 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-var-lib-openvswitch\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.221807 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220557 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-host-run-multus-certs\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.221807 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220571 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-host-run-k8s-cni-cncf-io\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.221807 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220580 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-hostroot\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.222727 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220579 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d8089437-f546-441e-b121-6e79878d2608-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tc87d\" (UID: \"d8089437-f546-441e-b121-6e79878d2608\") " pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:58:14.222727 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:14.220585 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:58:14.222727 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220605 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-host-var-lib-kubelet\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.222727 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220612 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-host-run-k8s-cni-cncf-io\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.222727 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220617 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-systemd-units\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.222727 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220631 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-slash\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.222727 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220681 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-slash\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.222727 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:14.220691 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0adc098d-57a4-490e-89b7-b1b271a1c087-metrics-certs podName:0adc098d-57a4-490e-89b7-b1b271a1c087 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:14.720648045 +0000 UTC m=+2.037797358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0adc098d-57a4-490e-89b7-b1b271a1c087-metrics-certs") pod "network-metrics-daemon-cjqfw" (UID: "0adc098d-57a4-490e-89b7-b1b271a1c087") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:58:14.222727 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220718 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-run-ovn-kubernetes\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.222727 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220750 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/63e868ff-3f36-4542-9e1d-9688ce8c37f1-etc-selinux\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:58:14.222727 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220765 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-run-ovn-kubernetes\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.222727 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220777 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-sysconfig\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.222727 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220805 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-run\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.222727 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220828 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-host\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.222727 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220723 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-host-var-lib-kubelet\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.222727 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220853 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5405bfc2-cfc0-4ea6-80bc-911e117f1843-tmp\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.222727 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220861 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/63e868ff-3f36-4542-9e1d-9688ce8c37f1-etc-selinux\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:58:14.223607 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220894 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-sysconfig\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.223607 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220913 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-run\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.223607 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220940 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a0be3213-788c-45b4-8fbd-37e744a14f17-cni-binary-copy\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.223607 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220964 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-run-systemd\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.223607 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.221000 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-cni-netd\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.223607 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.221020 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.223607 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.221050 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/25e99515-222f-4346-9fe5-5b7b3b213ee8-env-overrides\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.223607 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.221082 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8089437-f546-441e-b121-6e79878d2608-cni-binary-copy\") pod \"multus-additional-cni-plugins-tc87d\" (UID: \"d8089437-f546-441e-b121-6e79878d2608\") " pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:58:14.223607 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.221099 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/18acd0a8-988e-4168-84e7-0b25dc7b32d4-serviceca\") pod \"node-ca-vnk6j\" (UID: \"18acd0a8-988e-4168-84e7-0b25dc7b32d4\") " pod="openshift-image-registry/node-ca-vnk6j" Oct 30 23:58:14.223607 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.221106 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8wrl\" (UniqueName: \"kubernetes.io/projected/d8089437-f546-441e-b121-6e79878d2608-kube-api-access-d8wrl\") pod \"multus-additional-cni-plugins-tc87d\" (UID: \"d8089437-f546-441e-b121-6e79878d2608\") " pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:58:14.223607 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.221134 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/409b7296-1d35-44e9-be1a-3459f0dbaad9-agent-certs\") pod \"konnectivity-agent-x962k\" (UID: \"409b7296-1d35-44e9-be1a-3459f0dbaad9\") " pod="kube-system/konnectivity-agent-x962k" Oct 30 23:58:14.223607 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.221157 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-node-log\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.223607 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.221179 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-cni-bin\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.223607 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.221195 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.223607 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.221208 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl2s5\" (UniqueName: \"kubernetes.io/projected/63e868ff-3f36-4542-9e1d-9688ce8c37f1-kube-api-access-fl2s5\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:58:14.223607 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.221236 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-tuned\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.223607 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.221243 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-run-systemd\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.224130 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.221263 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-multus-cni-dir\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.224130 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.221291 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlqqp\" (UniqueName: \"kubernetes.io/projected/a0be3213-788c-45b4-8fbd-37e744a14f17-kube-api-access-hlqqp\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.224130 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.221304 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-cni-netd\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.224130 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.221317 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-run-openvswitch\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.224130 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.221320 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/25e99515-222f-4346-9fe5-5b7b3b213ee8-ovnkube-config\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.224130 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.221343 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-log-socket\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.224130 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.221351 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-node-log\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.224130 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.221358 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a0be3213-788c-45b4-8fbd-37e744a14f17-cni-binary-copy\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.224130 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.221369 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8089437-f546-441e-b121-6e79878d2608-cnibin\") pod \"multus-additional-cni-plugins-tc87d\" (UID: \"d8089437-f546-441e-b121-6e79878d2608\") " pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:58:14.224130 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.221421 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8089437-f546-441e-b121-6e79878d2608-cnibin\") pod \"multus-additional-cni-plugins-tc87d\" (UID: \"d8089437-f546-441e-b121-6e79878d2608\") " pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:58:14.224130 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.221492 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-multus-cni-dir\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.224130 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.221623 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-cni-bin\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.224130 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.221649 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/25e99515-222f-4346-9fe5-5b7b3b213ee8-env-overrides\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.224130 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.221744 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-run-openvswitch\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.224130 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.221790 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8089437-f546-441e-b121-6e79878d2608-cni-binary-copy\") pod \"multus-additional-cni-plugins-tc87d\" (UID: \"d8089437-f546-441e-b121-6e79878d2608\") " pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:58:14.224130 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.221797 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-log-socket\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.224130 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.220940 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-host\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.224130 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.221851 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99b4d92d-2003-4708-99da-a5f12cbb10e2-original-pull-secret\") pod \"global-pull-secret-syncer-6l8ht\" (UID: \"99b4d92d-2003-4708-99da-a5f12cbb10e2\") " pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:58:14.224668 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.221880 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/63e868ff-3f36-4542-9e1d-9688ce8c37f1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:58:14.224668 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.221903 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-sysctl-d\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.224668 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.221926 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-sys\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.224668 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.221950 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-var-lib-kubelet\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.224668 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.221974 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/409b7296-1d35-44e9-be1a-3459f0dbaad9-konnectivity-ca\") pod \"konnectivity-agent-x962k\" (UID: \"409b7296-1d35-44e9-be1a-3459f0dbaad9\") " pod="kube-system/konnectivity-agent-x962k" Oct 30 23:58:14.224668 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.221998 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/25e99515-222f-4346-9fe5-5b7b3b213ee8-ovnkube-script-lib\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.224668 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.221998 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/63e868ff-3f36-4542-9e1d-9688ce8c37f1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:58:14.224668 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.222037 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8089437-f546-441e-b121-6e79878d2608-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tc87d\" (UID: \"d8089437-f546-441e-b121-6e79878d2608\") " pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:58:14.224668 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.222067 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/63e868ff-3f36-4542-9e1d-9688ce8c37f1-device-dir\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:58:14.224668 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.222093 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djj7v\" (UniqueName: \"kubernetes.io/projected/0adc098d-57a4-490e-89b7-b1b271a1c087-kube-api-access-djj7v\") pod \"network-metrics-daemon-cjqfw\" (UID: \"0adc098d-57a4-490e-89b7-b1b271a1c087\") " pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:58:14.224668 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:14.222114 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:58:14.224668 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.222129 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-sysctl-d\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.224668 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:14.222189 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99b4d92d-2003-4708-99da-a5f12cbb10e2-original-pull-secret podName:99b4d92d-2003-4708-99da-a5f12cbb10e2 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:14.722169604 +0000 UTC m=+2.039318909 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99b4d92d-2003-4708-99da-a5f12cbb10e2-original-pull-secret") pod "global-pull-secret-syncer-6l8ht" (UID: "99b4d92d-2003-4708-99da-a5f12cbb10e2") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:58:14.224668 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.222316 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8089437-f546-441e-b121-6e79878d2608-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tc87d\" (UID: \"d8089437-f546-441e-b121-6e79878d2608\") " pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:58:14.224668 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.222559 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/63e868ff-3f36-4542-9e1d-9688ce8c37f1-device-dir\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:58:14.224668 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.222616 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-sys\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.224668 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.222666 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-var-lib-kubelet\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.225170 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.223185 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/409b7296-1d35-44e9-be1a-3459f0dbaad9-konnectivity-ca\") pod \"konnectivity-agent-x962k\" (UID: \"409b7296-1d35-44e9-be1a-3459f0dbaad9\") " pod="kube-system/konnectivity-agent-x962k" Oct 30 23:58:14.225170 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.223258 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/25e99515-222f-4346-9fe5-5b7b3b213ee8-ovnkube-script-lib\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.225170 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.223805 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/25e99515-222f-4346-9fe5-5b7b3b213ee8-ovn-node-metrics-cert\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.225170 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.223822 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-tuned\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.225170 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.224704 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/409b7296-1d35-44e9-be1a-3459f0dbaad9-agent-certs\") pod \"konnectivity-agent-x962k\" (UID: \"409b7296-1d35-44e9-be1a-3459f0dbaad9\") " pod="kube-system/konnectivity-agent-x962k" Oct 30 23:58:14.226258 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.226236 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5405bfc2-cfc0-4ea6-80bc-911e117f1843-tmp\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.232114 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.232093 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8tds\" (UniqueName: \"kubernetes.io/projected/1f653a44-907a-417e-a1da-9b1bb0ee7283-kube-api-access-r8tds\") pod \"iptables-alerter-zlgls\" (UID: \"1f653a44-907a-417e-a1da-9b1bb0ee7283\") " pod="openshift-network-operator/iptables-alerter-zlgls" Oct 30 23:58:14.235178 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.235158 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcvd6\" (UniqueName: \"kubernetes.io/projected/5405bfc2-cfc0-4ea6-80bc-911e117f1843-kube-api-access-mcvd6\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.236997 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:14.236978 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:58:14.237075 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:14.237003 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:58:14.237075 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:14.237066 2574 projected.go:196] Error preparing data for projected volume kube-api-access-gd4j2 for pod openshift-network-diagnostics/network-check-target-bd4bx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:58:14.237167 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:14.237154 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50c15c32-e5cd-426c-9e7d-c123f8480653-kube-api-access-gd4j2 podName:50c15c32-e5cd-426c-9e7d-c123f8480653 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:14.737136139 +0000 UTC m=+2.054285448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gd4j2" (UniqueName: "kubernetes.io/projected/50c15c32-e5cd-426c-9e7d-c123f8480653-kube-api-access-gd4j2") pod "network-check-target-bd4bx" (UID: "50c15c32-e5cd-426c-9e7d-c123f8480653") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:58:14.238785 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.238768 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmcgf\" (UniqueName: \"kubernetes.io/projected/18acd0a8-988e-4168-84e7-0b25dc7b32d4-kube-api-access-pmcgf\") pod \"node-ca-vnk6j\" (UID: \"18acd0a8-988e-4168-84e7-0b25dc7b32d4\") " pod="openshift-image-registry/node-ca-vnk6j" Oct 30 23:58:14.239059 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.239022 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpxpg\" (UniqueName: \"kubernetes.io/projected/25e99515-222f-4346-9fe5-5b7b3b213ee8-kube-api-access-dpxpg\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.241576 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.241552 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8wrl\" (UniqueName: \"kubernetes.io/projected/d8089437-f546-441e-b121-6e79878d2608-kube-api-access-d8wrl\") pod \"multus-additional-cni-plugins-tc87d\" (UID: \"d8089437-f546-441e-b121-6e79878d2608\") " pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:58:14.241875 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.241851 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djj7v\" (UniqueName: \"kubernetes.io/projected/0adc098d-57a4-490e-89b7-b1b271a1c087-kube-api-access-djj7v\") pod \"network-metrics-daemon-cjqfw\" (UID: \"0adc098d-57a4-490e-89b7-b1b271a1c087\") " pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:58:14.242141 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.242123 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl2s5\" (UniqueName: \"kubernetes.io/projected/63e868ff-3f36-4542-9e1d-9688ce8c37f1-kube-api-access-fl2s5\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:58:14.242293 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.242277 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlqqp\" (UniqueName: \"kubernetes.io/projected/a0be3213-788c-45b4-8fbd-37e744a14f17-kube-api-access-hlqqp\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.260631 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:14.260601 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e9a3bb1bd7fbf1e3c59a1731918cc24.slice/crio-34539fa69c119c636d1acc43e480f7606c57d5d847b5c124491b8ea368c84256 WatchSource:0}: Error finding container 34539fa69c119c636d1acc43e480f7606c57d5d847b5c124491b8ea368c84256: Status 404 returned error can't find the container with id 34539fa69c119c636d1acc43e480f7606c57d5d847b5c124491b8ea368c84256 Oct 30 23:58:14.261275 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:14.261260 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4b01b095f68d38f7d23560b244b5e01.slice/crio-a5b7fb8850839f327a21ed0ca1950d02333706f22908cae2ae7631d85e53f37b WatchSource:0}: Error finding container a5b7fb8850839f327a21ed0ca1950d02333706f22908cae2ae7631d85e53f37b: Status 404 returned error can't find the container with id a5b7fb8850839f327a21ed0ca1950d02333706f22908cae2ae7631d85e53f37b Oct 30 23:58:14.268174 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.268158 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 23:58:14.330651 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.330600 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6hq7b" Oct 30 23:58:14.339793 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:14.339772 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7815306_7a99_4fc5_8226_b34916ceaaa2.slice/crio-8bf9deef04f1f887138a935ae7aa51b824efe2c52092d0c3d2111a2af206d09d WatchSource:0}: Error finding container 8bf9deef04f1f887138a935ae7aa51b824efe2c52092d0c3d2111a2af206d09d: Status 404 returned error can't find the container with id 8bf9deef04f1f887138a935ae7aa51b824efe2c52092d0c3d2111a2af206d09d Oct 30 23:58:14.350267 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.350250 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zlgls" Oct 30 23:58:14.358414 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:14.358396 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f653a44_907a_417e_a1da_9b1bb0ee7283.slice/crio-2c4dfef2498ce2f91228a57d0ac1e3a8e6124e8c5c0694400179b595b0aade48 WatchSource:0}: Error finding container 2c4dfef2498ce2f91228a57d0ac1e3a8e6124e8c5c0694400179b595b0aade48: Status 404 returned error can't find the container with id 2c4dfef2498ce2f91228a57d0ac1e3a8e6124e8c5c0694400179b595b0aade48 Oct 30 23:58:14.382121 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.382103 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:58:14.384441 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.384402 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vnk6j" Oct 30 23:58:14.390905 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:14.390881 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8089437_f546_441e_b121_6e79878d2608.slice/crio-04a789b9af69c8958d287e296d45a9a03c561c67ef9faed9c7ab80d450c456c9 WatchSource:0}: Error finding container 04a789b9af69c8958d287e296d45a9a03c561c67ef9faed9c7ab80d450c456c9: Status 404 returned error can't find the container with id 04a789b9af69c8958d287e296d45a9a03c561c67ef9faed9c7ab80d450c456c9 Oct 30 23:58:14.392875 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:14.392855 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18acd0a8_988e_4168_84e7_0b25dc7b32d4.slice/crio-dbe38159a2b22ae460f368a9a9408a07e42e1078e30811ab06e7c16edd791eb9 WatchSource:0}: Error finding container dbe38159a2b22ae460f368a9a9408a07e42e1078e30811ab06e7c16edd791eb9: Status 404 returned error can't find the container with id dbe38159a2b22ae460f368a9a9408a07e42e1078e30811ab06e7c16edd791eb9 Oct 30 23:58:14.397280 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.397260 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9j82q" Oct 30 23:58:14.406115 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:14.406089 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0be3213_788c_45b4_8fbd_37e744a14f17.slice/crio-a01b5325ff7f82a3e316636bffb553afc1f5fa152a18c4ec81a031759ec90dad WatchSource:0}: Error finding container a01b5325ff7f82a3e316636bffb553afc1f5fa152a18c4ec81a031759ec90dad: Status 404 returned error can't find the container with id a01b5325ff7f82a3e316636bffb553afc1f5fa152a18c4ec81a031759ec90dad Oct 30 23:58:14.417916 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.417898 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x962k" Oct 30 23:58:14.427345 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:14.427321 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod409b7296_1d35_44e9_be1a_3459f0dbaad9.slice/crio-b3dd912a576bc95f147b1188d016ea64820726576094e8de17e26a806ea21706 WatchSource:0}: Error finding container b3dd912a576bc95f147b1188d016ea64820726576094e8de17e26a806ea21706: Status 404 returned error can't find the container with id b3dd912a576bc95f147b1188d016ea64820726576094e8de17e26a806ea21706 Oct 30 23:58:14.442170 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.442154 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:58:14.451477 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:14.451454 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63e868ff_3f36_4542_9e1d_9688ce8c37f1.slice/crio-92470d7928996059c02f744bd144c8e341b169b6fc78e28dcb0746dd804aee25 WatchSource:0}: Error finding container 92470d7928996059c02f744bd144c8e341b169b6fc78e28dcb0746dd804aee25: Status 404 returned error can't find the container with id 92470d7928996059c02f744bd144c8e341b169b6fc78e28dcb0746dd804aee25 Oct 30 23:58:14.460439 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.460423 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:14.462549 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.462534 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:58:14.472580 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:14.472560 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25e99515_222f_4346_9fe5_5b7b3b213ee8.slice/crio-7ed7577cf3a690a028a8810df75503f3a7aaa9b5ee22832c2944fd235487c84e WatchSource:0}: Error finding container 7ed7577cf3a690a028a8810df75503f3a7aaa9b5ee22832c2944fd235487c84e: Status 404 returned error can't find the container with id 7ed7577cf3a690a028a8810df75503f3a7aaa9b5ee22832c2944fd235487c84e Oct 30 23:58:14.475284 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:58:14.475257 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5405bfc2_cfc0_4ea6_80bc_911e117f1843.slice/crio-5629ab8e8b75611da6148352576cdf691dea914e4830af7fc1830a99ab58b14e WatchSource:0}: Error finding container 5629ab8e8b75611da6148352576cdf691dea914e4830af7fc1830a99ab58b14e: Status 404 returned error can't find the container with id 5629ab8e8b75611da6148352576cdf691dea914e4830af7fc1830a99ab58b14e Oct 30 23:58:14.595288 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.595238 2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:58:14.725378 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.725358 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0adc098d-57a4-490e-89b7-b1b271a1c087-metrics-certs\") pod \"network-metrics-daemon-cjqfw\" (UID: \"0adc098d-57a4-490e-89b7-b1b271a1c087\") " pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:58:14.725471 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.725401 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99b4d92d-2003-4708-99da-a5f12cbb10e2-original-pull-secret\") pod \"global-pull-secret-syncer-6l8ht\" (UID: \"99b4d92d-2003-4708-99da-a5f12cbb10e2\") " pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:58:14.725513 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:14.725490 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:58:14.725513 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:14.725491 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:58:14.725573 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:14.725533 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99b4d92d-2003-4708-99da-a5f12cbb10e2-original-pull-secret podName:99b4d92d-2003-4708-99da-a5f12cbb10e2 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:15.725521194 +0000 UTC m=+3.042670486 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99b4d92d-2003-4708-99da-a5f12cbb10e2-original-pull-secret") pod "global-pull-secret-syncer-6l8ht" (UID: "99b4d92d-2003-4708-99da-a5f12cbb10e2") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:58:14.725573 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:14.725549 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0adc098d-57a4-490e-89b7-b1b271a1c087-metrics-certs podName:0adc098d-57a4-490e-89b7-b1b271a1c087 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:15.725540325 +0000 UTC m=+3.042689617 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0adc098d-57a4-490e-89b7-b1b271a1c087-metrics-certs") pod "network-metrics-daemon-cjqfw" (UID: "0adc098d-57a4-490e-89b7-b1b271a1c087") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:58:14.825850 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:14.825820 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd4j2\" (UniqueName: \"kubernetes.io/projected/50c15c32-e5cd-426c-9e7d-c123f8480653-kube-api-access-gd4j2\") pod \"network-check-target-bd4bx\" (UID: \"50c15c32-e5cd-426c-9e7d-c123f8480653\") " pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:58:14.825981 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:14.825958 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:58:14.826123 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:14.825996 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:58:14.826123 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:14.826009 2574 projected.go:196] Error preparing data for projected volume kube-api-access-gd4j2 for pod openshift-network-diagnostics/network-check-target-bd4bx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:58:14.826123 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:14.826065 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50c15c32-e5cd-426c-9e7d-c123f8480653-kube-api-access-gd4j2 podName:50c15c32-e5cd-426c-9e7d-c123f8480653 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:15.826046183 +0000 UTC m=+3.143195495 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gd4j2" (UniqueName: "kubernetes.io/projected/50c15c32-e5cd-426c-9e7d-c123f8480653-kube-api-access-gd4j2") pod "network-check-target-bd4bx" (UID: "50c15c32-e5cd-426c-9e7d-c123f8480653") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:58:15.090911 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:15.090786 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:53:14 +0000 UTC" deadline="2028-08-05 02:25:00.770781656 +0000 UTC" Oct 30 23:58:15.090911 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:15.090847 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15458h26m45.679938612s" Oct 30 23:58:15.165227 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:15.165196 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:58:15.165412 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:15.165371 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4bx" podUID="50c15c32-e5cd-426c-9e7d-c123f8480653" Oct 30 23:58:15.165923 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:15.165901 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:58:15.166149 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:15.166105 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjqfw" podUID="0adc098d-57a4-490e-89b7-b1b271a1c087" Oct 30 23:58:15.198046 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:15.197870 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f64c8" event={"ID":"5405bfc2-cfc0-4ea6-80bc-911e117f1843","Type":"ContainerStarted","Data":"5629ab8e8b75611da6148352576cdf691dea914e4830af7fc1830a99ab58b14e"} Oct 30 23:58:15.213131 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:15.213034 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" event={"ID":"63e868ff-3f36-4542-9e1d-9688ce8c37f1","Type":"ContainerStarted","Data":"92470d7928996059c02f744bd144c8e341b169b6fc78e28dcb0746dd804aee25"} Oct 30 23:58:15.237142 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:15.237105 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9j82q" event={"ID":"a0be3213-788c-45b4-8fbd-37e744a14f17","Type":"ContainerStarted","Data":"a01b5325ff7f82a3e316636bffb553afc1f5fa152a18c4ec81a031759ec90dad"} Oct 30 23:58:15.251445 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:15.251408 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vnk6j" event={"ID":"18acd0a8-988e-4168-84e7-0b25dc7b32d4","Type":"ContainerStarted","Data":"dbe38159a2b22ae460f368a9a9408a07e42e1078e30811ab06e7c16edd791eb9"} Oct 30 23:58:15.256183 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:15.256155 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zlgls" event={"ID":"1f653a44-907a-417e-a1da-9b1bb0ee7283","Type":"ContainerStarted","Data":"2c4dfef2498ce2f91228a57d0ac1e3a8e6124e8c5c0694400179b595b0aade48"} Oct 30 23:58:15.268402 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:15.268374 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6hq7b" event={"ID":"d7815306-7a99-4fc5-8226-b34916ceaaa2","Type":"ContainerStarted","Data":"8bf9deef04f1f887138a935ae7aa51b824efe2c52092d0c3d2111a2af206d09d"} Oct 30 23:58:15.285195 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:15.285090 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" event={"ID":"25e99515-222f-4346-9fe5-5b7b3b213ee8","Type":"ContainerStarted","Data":"7ed7577cf3a690a028a8810df75503f3a7aaa9b5ee22832c2944fd235487c84e"} Oct 30 23:58:15.296292 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:15.296173 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x962k" event={"ID":"409b7296-1d35-44e9-be1a-3459f0dbaad9","Type":"ContainerStarted","Data":"b3dd912a576bc95f147b1188d016ea64820726576094e8de17e26a806ea21706"} Oct 30 23:58:15.313018 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:15.312943 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tc87d" event={"ID":"d8089437-f546-441e-b121-6e79878d2608","Type":"ContainerStarted","Data":"04a789b9af69c8958d287e296d45a9a03c561c67ef9faed9c7ab80d450c456c9"} Oct 30 23:58:15.326772 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:15.326650 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal" event={"ID":"c4b01b095f68d38f7d23560b244b5e01","Type":"ContainerStarted","Data":"a5b7fb8850839f327a21ed0ca1950d02333706f22908cae2ae7631d85e53f37b"} Oct 30 23:58:15.332344 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:15.332290 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-70.us-east-2.compute.internal" event={"ID":"2e9a3bb1bd7fbf1e3c59a1731918cc24","Type":"ContainerStarted","Data":"34539fa69c119c636d1acc43e480f7606c57d5d847b5c124491b8ea368c84256"} Oct 30 23:58:15.392661 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:15.392566 2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:58:15.735047 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:15.734952 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:58:15.735223 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:15.735053 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99b4d92d-2003-4708-99da-a5f12cbb10e2-original-pull-secret podName:99b4d92d-2003-4708-99da-a5f12cbb10e2 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:17.73503127 +0000 UTC m=+5.052180564 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99b4d92d-2003-4708-99da-a5f12cbb10e2-original-pull-secret") pod "global-pull-secret-syncer-6l8ht" (UID: "99b4d92d-2003-4708-99da-a5f12cbb10e2") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:58:15.735597 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:15.734602 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99b4d92d-2003-4708-99da-a5f12cbb10e2-original-pull-secret\") pod \"global-pull-secret-syncer-6l8ht\" (UID: \"99b4d92d-2003-4708-99da-a5f12cbb10e2\") " pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:58:15.735744 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:15.735725 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0adc098d-57a4-490e-89b7-b1b271a1c087-metrics-certs\") pod \"network-metrics-daemon-cjqfw\" (UID: \"0adc098d-57a4-490e-89b7-b1b271a1c087\") " pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:58:15.735863 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:15.735842 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:58:15.735927 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:15.735897 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0adc098d-57a4-490e-89b7-b1b271a1c087-metrics-certs podName:0adc098d-57a4-490e-89b7-b1b271a1c087 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:17.735880424 +0000 UTC m=+5.053029717 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0adc098d-57a4-490e-89b7-b1b271a1c087-metrics-certs") pod "network-metrics-daemon-cjqfw" (UID: "0adc098d-57a4-490e-89b7-b1b271a1c087") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:58:15.836835 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:15.836799 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd4j2\" (UniqueName: \"kubernetes.io/projected/50c15c32-e5cd-426c-9e7d-c123f8480653-kube-api-access-gd4j2\") pod \"network-check-target-bd4bx\" (UID: \"50c15c32-e5cd-426c-9e7d-c123f8480653\") " pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:58:15.837022 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:15.836987 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:58:15.837022 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:15.837032 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:58:15.837192 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:15.837052 2574 projected.go:196] Error preparing data for projected volume kube-api-access-gd4j2 for pod openshift-network-diagnostics/network-check-target-bd4bx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:58:15.837192 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:15.837106 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50c15c32-e5cd-426c-9e7d-c123f8480653-kube-api-access-gd4j2 podName:50c15c32-e5cd-426c-9e7d-c123f8480653 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:17.837089758 +0000 UTC m=+5.154239053 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gd4j2" (UniqueName: "kubernetes.io/projected/50c15c32-e5cd-426c-9e7d-c123f8480653-kube-api-access-gd4j2") pod "network-check-target-bd4bx" (UID: "50c15c32-e5cd-426c-9e7d-c123f8480653") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:58:16.091448 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:16.091331 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:53:14 +0000 UTC" deadline="2028-07-11 21:59:42.772446948 +0000 UTC" Oct 30 23:58:16.091448 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:16.091399 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14878h1m26.681052594s" Oct 30 23:58:16.154135 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:16.154107 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:58:16.154309 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:16.154246 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6l8ht" podUID="99b4d92d-2003-4708-99da-a5f12cbb10e2" Oct 30 23:58:16.494717 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:16.494449 2574 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:58:17.157050 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:17.154593 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:58:17.157050 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:17.154728 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4bx" podUID="50c15c32-e5cd-426c-9e7d-c123f8480653" Oct 30 23:58:17.157050 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:17.155242 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:58:17.157050 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:17.155355 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjqfw" podUID="0adc098d-57a4-490e-89b7-b1b271a1c087" Oct 30 23:58:17.755728 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:17.754909 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99b4d92d-2003-4708-99da-a5f12cbb10e2-original-pull-secret\") pod \"global-pull-secret-syncer-6l8ht\" (UID: \"99b4d92d-2003-4708-99da-a5f12cbb10e2\") " pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:58:17.755728 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:17.754986 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0adc098d-57a4-490e-89b7-b1b271a1c087-metrics-certs\") pod \"network-metrics-daemon-cjqfw\" (UID: \"0adc098d-57a4-490e-89b7-b1b271a1c087\") " pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:58:17.755728 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:17.755122 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:58:17.755728 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:17.755184 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0adc098d-57a4-490e-89b7-b1b271a1c087-metrics-certs podName:0adc098d-57a4-490e-89b7-b1b271a1c087 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:21.755164006 +0000 UTC m=+9.072313316 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0adc098d-57a4-490e-89b7-b1b271a1c087-metrics-certs") pod "network-metrics-daemon-cjqfw" (UID: "0adc098d-57a4-490e-89b7-b1b271a1c087") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:58:17.755728 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:17.755581 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:58:17.756235 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:17.756213 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99b4d92d-2003-4708-99da-a5f12cbb10e2-original-pull-secret podName:99b4d92d-2003-4708-99da-a5f12cbb10e2 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:21.755650988 +0000 UTC m=+9.072800280 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99b4d92d-2003-4708-99da-a5f12cbb10e2-original-pull-secret") pod "global-pull-secret-syncer-6l8ht" (UID: "99b4d92d-2003-4708-99da-a5f12cbb10e2") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:58:17.856811 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:17.856676 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd4j2\" (UniqueName: \"kubernetes.io/projected/50c15c32-e5cd-426c-9e7d-c123f8480653-kube-api-access-gd4j2\") pod \"network-check-target-bd4bx\" (UID: \"50c15c32-e5cd-426c-9e7d-c123f8480653\") " pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:58:17.856996 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:17.856882 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:58:17.856996 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:17.856906 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:58:17.856996 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:17.856919 2574 projected.go:196] Error preparing data for projected volume kube-api-access-gd4j2 for pod openshift-network-diagnostics/network-check-target-bd4bx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:58:17.856996 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:17.856974 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50c15c32-e5cd-426c-9e7d-c123f8480653-kube-api-access-gd4j2 podName:50c15c32-e5cd-426c-9e7d-c123f8480653 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:21.856955242 +0000 UTC m=+9.174104539 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gd4j2" (UniqueName: "kubernetes.io/projected/50c15c32-e5cd-426c-9e7d-c123f8480653-kube-api-access-gd4j2") pod "network-check-target-bd4bx" (UID: "50c15c32-e5cd-426c-9e7d-c123f8480653") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:58:18.153897 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:18.153812 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:58:18.154056 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:18.153937 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6l8ht" podUID="99b4d92d-2003-4708-99da-a5f12cbb10e2" Oct 30 23:58:19.157939 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:19.156268 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:58:19.157939 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:19.156647 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4bx" podUID="50c15c32-e5cd-426c-9e7d-c123f8480653" Oct 30 23:58:19.157939 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:19.157068 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:58:19.157939 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:19.157184 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjqfw" podUID="0adc098d-57a4-490e-89b7-b1b271a1c087" Oct 30 23:58:19.361192 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:19.361073 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x962k" event={"ID":"409b7296-1d35-44e9-be1a-3459f0dbaad9","Type":"ContainerStarted","Data":"4f8ae668f098dd043ff18120c3b4c17db6333b109ee9f07017fa01ab5804f95d"} Oct 30 23:58:19.370206 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:19.370172 2574 generic.go:356] "Generic (PLEG): container finished" podID="d8089437-f546-441e-b121-6e79878d2608" containerID="fd14f9bd44e7d8b2299bc7aaf2e6102a6ec274b31046a99b3ceb1d4cb24dc718" exitCode=0 Oct 30 23:58:19.370495 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:19.370468 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tc87d" event={"ID":"d8089437-f546-441e-b121-6e79878d2608","Type":"ContainerDied","Data":"fd14f9bd44e7d8b2299bc7aaf2e6102a6ec274b31046a99b3ceb1d4cb24dc718"} Oct 30 23:58:19.382320 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:19.380535 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-70.us-east-2.compute.internal" event={"ID":"2e9a3bb1bd7fbf1e3c59a1731918cc24","Type":"ContainerStarted","Data":"41b94ee11f573ef72e0c10cb0486a9b0b00e8801cebce2f917c72407c6d3c8ed"} Oct 30 23:58:19.383474 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:19.383423 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-x962k" podStartSLOduration=0.772158199 podStartE2EDuration="5.38340626s" totalImagesPullingTime="4.611248061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:19.382799456 +0000 UTC m=+6.699948774" watchObservedRunningTime="2026-10-30 23:58:19.38340626 +0000 UTC m=+6.700555577" Oct 30 23:58:19.441489 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:19.440175 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-70.us-east-2.compute.internal" podStartSLOduration=5.440156007 podStartE2EDuration="5.440156007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:58:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:19.413251392 +0000 UTC m=+6.730400707" watchObservedRunningTime="2026-10-30 23:58:19.440156007 +0000 UTC m=+6.757305325" Oct 30 23:58:20.154865 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:20.154255 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:58:20.154865 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:20.154395 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6l8ht" podUID="99b4d92d-2003-4708-99da-a5f12cbb10e2" Oct 30 23:58:20.391786 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:20.389135 2574 generic.go:356] "Generic (PLEG): container finished" podID="c4b01b095f68d38f7d23560b244b5e01" containerID="d2013d77d668064652b566a512f5446c43c712219e78c1d3847a2acea0b1b2cc" exitCode=0 Oct 30 23:58:20.391786 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:20.390264 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal" event={"ID":"c4b01b095f68d38f7d23560b244b5e01","Type":"ContainerDied","Data":"d2013d77d668064652b566a512f5446c43c712219e78c1d3847a2acea0b1b2cc"} Oct 30 23:58:21.155244 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:21.155160 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:58:21.155490 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:21.155359 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4bx" podUID="50c15c32-e5cd-426c-9e7d-c123f8480653" Oct 30 23:58:21.155744 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:21.155724 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:58:21.155872 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:21.155851 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjqfw" podUID="0adc098d-57a4-490e-89b7-b1b271a1c087" Oct 30 23:58:21.398820 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:21.398784 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" event={"ID":"63e868ff-3f36-4542-9e1d-9688ce8c37f1","Type":"ContainerStarted","Data":"ba1104832ceee3e740061e098414e36f13ff43ae01e09f2fbf757e9a2a45c183"} Oct 30 23:58:21.402496 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:21.402464 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vnk6j" event={"ID":"18acd0a8-988e-4168-84e7-0b25dc7b32d4","Type":"ContainerStarted","Data":"20ff75eca731500e211113c299d87bc425511a9cbfdc0ffe9bbda7de355970c1"} Oct 30 23:58:21.405805 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:21.405711 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal" event={"ID":"c4b01b095f68d38f7d23560b244b5e01","Type":"ContainerStarted","Data":"0d873c115b99713928439afe81106a9c4306a9334bad5b00fc54ad3c142c3519"} Oct 30 23:58:21.453281 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:21.453218 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vnk6j" podStartSLOduration=1.456699928 podStartE2EDuration="7.452833448s" totalImagesPullingTime="5.99613352s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:58:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:21.425199237 +0000 UTC m=+8.742348553" watchObservedRunningTime="2026-10-30 23:58:21.452833448 +0000 UTC m=+8.769982763" Oct 30 23:58:21.790899 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:21.790846 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99b4d92d-2003-4708-99da-a5f12cbb10e2-original-pull-secret\") pod \"global-pull-secret-syncer-6l8ht\" (UID: \"99b4d92d-2003-4708-99da-a5f12cbb10e2\") " pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:58:21.791110 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:21.790932 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0adc098d-57a4-490e-89b7-b1b271a1c087-metrics-certs\") pod \"network-metrics-daemon-cjqfw\" (UID: \"0adc098d-57a4-490e-89b7-b1b271a1c087\") " pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:58:21.791110 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:21.791089 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:58:21.791268 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:21.791151 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0adc098d-57a4-490e-89b7-b1b271a1c087-metrics-certs podName:0adc098d-57a4-490e-89b7-b1b271a1c087 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:29.791131496 +0000 UTC m=+17.108280804 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0adc098d-57a4-490e-89b7-b1b271a1c087-metrics-certs") pod "network-metrics-daemon-cjqfw" (UID: "0adc098d-57a4-490e-89b7-b1b271a1c087") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:58:21.791530 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:21.791509 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:58:21.791630 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:21.791552 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99b4d92d-2003-4708-99da-a5f12cbb10e2-original-pull-secret podName:99b4d92d-2003-4708-99da-a5f12cbb10e2 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:29.79154117 +0000 UTC m=+17.108690480 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99b4d92d-2003-4708-99da-a5f12cbb10e2-original-pull-secret") pod "global-pull-secret-syncer-6l8ht" (UID: "99b4d92d-2003-4708-99da-a5f12cbb10e2") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:58:21.891536 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:21.891436 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd4j2\" (UniqueName: \"kubernetes.io/projected/50c15c32-e5cd-426c-9e7d-c123f8480653-kube-api-access-gd4j2\") pod \"network-check-target-bd4bx\" (UID: \"50c15c32-e5cd-426c-9e7d-c123f8480653\") " pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:58:21.891728 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:21.891629 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:58:21.891728 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:21.891652 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:58:21.891728 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:21.891666 2574 projected.go:196] Error preparing data for projected volume kube-api-access-gd4j2 for pod openshift-network-diagnostics/network-check-target-bd4bx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:58:21.891728 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:21.891726 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50c15c32-e5cd-426c-9e7d-c123f8480653-kube-api-access-gd4j2 podName:50c15c32-e5cd-426c-9e7d-c123f8480653 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:29.891706089 +0000 UTC m=+17.208855401 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gd4j2" (UniqueName: "kubernetes.io/projected/50c15c32-e5cd-426c-9e7d-c123f8480653-kube-api-access-gd4j2") pod "network-check-target-bd4bx" (UID: "50c15c32-e5cd-426c-9e7d-c123f8480653") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:58:22.153772 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:22.153692 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:58:22.153936 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:22.153830 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6l8ht" podUID="99b4d92d-2003-4708-99da-a5f12cbb10e2" Oct 30 23:58:22.739632 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:22.739463 2574 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 23:58:23.067500 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:23.067380 2574 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T23:58:22.739481343Z","UUID":"864962fc-43fe-4175-8b5d-c378f0b0197a","Handler":null,"Name":"","Endpoint":""} Oct 30 23:58:23.074578 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:23.074205 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 23:58:23.074578 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:23.074239 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 23:58:23.156824 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:23.155097 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:58:23.156824 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:23.155209 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjqfw" podUID="0adc098d-57a4-490e-89b7-b1b271a1c087" Oct 30 23:58:23.156824 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:23.155261 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:58:23.156824 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:23.155368 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4bx" podUID="50c15c32-e5cd-426c-9e7d-c123f8480653" Oct 30 23:58:23.414669 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:23.414635 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" event={"ID":"63e868ff-3f36-4542-9e1d-9688ce8c37f1","Type":"ContainerStarted","Data":"b52bce2e97d684a86091fa9f35fd3af7954e313b2dda2375a8959f14e4f5b957"} Oct 30 23:58:24.153828 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:24.153789 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:58:24.154303 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:24.153907 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6l8ht" podUID="99b4d92d-2003-4708-99da-a5f12cbb10e2" Oct 30 23:58:24.158033 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:24.157994 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-x962k" Oct 30 23:58:24.158697 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:24.158660 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-x962k" Oct 30 23:58:24.185224 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:24.185134 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal" podStartSLOduration=10.185104747 podStartE2EDuration="10.185104747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:58:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:21.453638395 +0000 UTC m=+8.770787687" watchObservedRunningTime="2026-10-30 23:58:24.185104747 +0000 UTC m=+11.502254057" Oct 30 23:58:24.418825 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:24.418790 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9j82q" event={"ID":"a0be3213-788c-45b4-8fbd-37e744a14f17","Type":"ContainerStarted","Data":"544c90c99bc0d348db2b7dce0e6464d4ff45a88fd71d2783115d2c8169eb4f4e"} Oct 30 23:58:24.420600 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:24.420561 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6hq7b" event={"ID":"d7815306-7a99-4fc5-8226-b34916ceaaa2","Type":"ContainerStarted","Data":"1edc40c3588008d69687a5e043daafef64900390b95130ec8a6ea9bb7affc91f"} Oct 30 23:58:24.422545 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:24.422517 2574 generic.go:356] "Generic (PLEG): container finished" podID="d8089437-f546-441e-b121-6e79878d2608" containerID="0aa944992685d16c0edc63b91222b665fd4fd75181c5c310834b3b9f8685c224" exitCode=0 Oct 30 23:58:24.422680 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:24.422606 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tc87d" event={"ID":"d8089437-f546-441e-b121-6e79878d2608","Type":"ContainerDied","Data":"0aa944992685d16c0edc63b91222b665fd4fd75181c5c310834b3b9f8685c224"} Oct 30 23:58:24.425519 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:24.425113 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" event={"ID":"63e868ff-3f36-4542-9e1d-9688ce8c37f1","Type":"ContainerStarted","Data":"78866afd858bcc890af4565f6c07686361d737a8a0cb9a1288e8221a7748f7f4"} Oct 30 23:58:24.446085 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:24.446015 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-9j82q" podStartSLOduration=1.044920523 podStartE2EDuration="10.446001084s" totalImagesPullingTime="9.401080561s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:58:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:24.445922167 +0000 UTC m=+11.763071482" watchObservedRunningTime="2026-10-30 23:58:24.446001084 +0000 UTC m=+11.763150396" Oct 30 23:58:24.524212 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:24.523986 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-6hq7b" podStartSLOduration=2.6271754019999998 podStartE2EDuration="11.523970832s" totalImagesPullingTime="8.89679543s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:58:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:24.490826958 +0000 UTC m=+11.807976273" watchObservedRunningTime="2026-10-30 23:58:24.523970832 +0000 UTC m=+11.841120149" Oct 30 23:58:24.524463 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:24.524420 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" podStartSLOduration=1.431894246 podStartE2EDuration="10.52440299s" totalImagesPullingTime="9.092508744s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:58:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:24.523931968 +0000 UTC m=+11.841081285" watchObservedRunningTime="2026-10-30 23:58:24.52440299 +0000 UTC m=+11.841552307" Oct 30 23:58:25.153916 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:25.153895 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:58:25.154346 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:25.153925 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:58:25.154346 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:25.154027 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4bx" podUID="50c15c32-e5cd-426c-9e7d-c123f8480653" Oct 30 23:58:25.154346 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:25.154146 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjqfw" podUID="0adc098d-57a4-490e-89b7-b1b271a1c087" Oct 30 23:58:25.428768 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:25.428701 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zlgls" event={"ID":"1f653a44-907a-417e-a1da-9b1bb0ee7283","Type":"ContainerStarted","Data":"23b142a3b8a0c84963a1e72277fbbff75182189f9ed82af0ab28e590ed0e191a"} Oct 30 23:58:25.430690 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:25.430669 2574 generic.go:356] "Generic (PLEG): container finished" podID="d8089437-f546-441e-b121-6e79878d2608" containerID="912e17c20f5cc26bd9e7e77ddfef14466a60d2601e36aee8401c92b5c26bfc4f" exitCode=0 Oct 30 23:58:25.430792 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:25.430749 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tc87d" event={"ID":"d8089437-f546-441e-b121-6e79878d2608","Type":"ContainerDied","Data":"912e17c20f5cc26bd9e7e77ddfef14466a60d2601e36aee8401c92b5c26bfc4f"} Oct 30 23:58:25.432182 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:25.432156 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f64c8" event={"ID":"5405bfc2-cfc0-4ea6-80bc-911e117f1843","Type":"ContainerStarted","Data":"394ca293fd1e22a00bbb4de5d63008aa84b15d64b17b389553a5ab038c4eea25"} Oct 30 23:58:25.447357 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:25.447318 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zlgls" podStartSLOduration=3.8085765670000002 podStartE2EDuration="12.447306423s" totalImagesPullingTime="8.638729856s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:58:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:25.447239154 +0000 UTC m=+12.764388463" watchObservedRunningTime="2026-10-30 23:58:25.447306423 +0000 UTC m=+12.764455735" Oct 30 23:58:25.495474 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:25.495440 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-f64c8" podStartSLOduration=0.953745129 podStartE2EDuration="11.495429462s" totalImagesPullingTime="10.541684333s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:58:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:25.495241909 +0000 UTC m=+12.812391221" watchObservedRunningTime="2026-10-30 23:58:25.495429462 +0000 UTC m=+12.812578775" Oct 30 23:58:26.153905 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:26.153876 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:58:26.154098 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:26.154040 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6l8ht" podUID="99b4d92d-2003-4708-99da-a5f12cbb10e2" Oct 30 23:58:26.436351 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:26.436324 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tc87d" event={"ID":"d8089437-f546-441e-b121-6e79878d2608","Type":"ContainerStarted","Data":"6daef47f6f360f0ba9f06ca03e6ca549bcbd61c14a8004c7aa81244c036ab7e7"} Oct 30 23:58:27.153708 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:27.153676 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:58:27.153864 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:27.153792 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4bx" podUID="50c15c32-e5cd-426c-9e7d-c123f8480653" Oct 30 23:58:27.153864 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:27.153844 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:58:27.153986 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:27.153968 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjqfw" podUID="0adc098d-57a4-490e-89b7-b1b271a1c087" Oct 30 23:58:27.440762 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:27.440691 2574 generic.go:356] "Generic (PLEG): container finished" podID="d8089437-f546-441e-b121-6e79878d2608" containerID="6daef47f6f360f0ba9f06ca03e6ca549bcbd61c14a8004c7aa81244c036ab7e7" exitCode=0 Oct 30 23:58:27.440762 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:27.440740 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tc87d" event={"ID":"d8089437-f546-441e-b121-6e79878d2608","Type":"ContainerDied","Data":"6daef47f6f360f0ba9f06ca03e6ca549bcbd61c14a8004c7aa81244c036ab7e7"} Oct 30 23:58:28.153982 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:28.153684 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:58:28.153982 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:28.153794 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6l8ht" podUID="99b4d92d-2003-4708-99da-a5f12cbb10e2" Oct 30 23:58:29.153867 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:29.153693 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:58:29.154348 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:29.153718 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:58:29.154348 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:29.153989 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4bx" podUID="50c15c32-e5cd-426c-9e7d-c123f8480653" Oct 30 23:58:29.154348 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:29.154050 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjqfw" podUID="0adc098d-57a4-490e-89b7-b1b271a1c087" Oct 30 23:58:29.863685 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:29.863656 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99b4d92d-2003-4708-99da-a5f12cbb10e2-original-pull-secret\") pod \"global-pull-secret-syncer-6l8ht\" (UID: \"99b4d92d-2003-4708-99da-a5f12cbb10e2\") " pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:58:29.863851 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:29.863710 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0adc098d-57a4-490e-89b7-b1b271a1c087-metrics-certs\") pod \"network-metrics-daemon-cjqfw\" (UID: \"0adc098d-57a4-490e-89b7-b1b271a1c087\") " pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:58:29.863851 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:29.863793 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:58:29.863851 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:29.863794 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:58:29.863851 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:29.863840 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0adc098d-57a4-490e-89b7-b1b271a1c087-metrics-certs podName:0adc098d-57a4-490e-89b7-b1b271a1c087 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:45.863826593 +0000 UTC m=+33.180975895 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0adc098d-57a4-490e-89b7-b1b271a1c087-metrics-certs") pod "network-metrics-daemon-cjqfw" (UID: "0adc098d-57a4-490e-89b7-b1b271a1c087") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:58:29.863851 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:29.863852 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99b4d92d-2003-4708-99da-a5f12cbb10e2-original-pull-secret podName:99b4d92d-2003-4708-99da-a5f12cbb10e2 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:45.863846746 +0000 UTC m=+33.180996037 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99b4d92d-2003-4708-99da-a5f12cbb10e2-original-pull-secret") pod "global-pull-secret-syncer-6l8ht" (UID: "99b4d92d-2003-4708-99da-a5f12cbb10e2") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:58:29.964136 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:29.964115 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd4j2\" (UniqueName: \"kubernetes.io/projected/50c15c32-e5cd-426c-9e7d-c123f8480653-kube-api-access-gd4j2\") pod \"network-check-target-bd4bx\" (UID: \"50c15c32-e5cd-426c-9e7d-c123f8480653\") " pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:58:29.964274 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:29.964251 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:58:29.964274 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:29.964268 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:58:29.964333 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:29.964277 2574 projected.go:196] Error preparing data for projected volume kube-api-access-gd4j2 for pod openshift-network-diagnostics/network-check-target-bd4bx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:58:29.964333 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:29.964316 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50c15c32-e5cd-426c-9e7d-c123f8480653-kube-api-access-gd4j2 podName:50c15c32-e5cd-426c-9e7d-c123f8480653 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:45.964304689 +0000 UTC m=+33.281453979 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gd4j2" (UniqueName: "kubernetes.io/projected/50c15c32-e5cd-426c-9e7d-c123f8480653-kube-api-access-gd4j2") pod "network-check-target-bd4bx" (UID: "50c15c32-e5cd-426c-9e7d-c123f8480653") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:58:30.153850 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:30.153824 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:58:30.153968 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:30.153926 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6l8ht" podUID="99b4d92d-2003-4708-99da-a5f12cbb10e2" Oct 30 23:58:30.449528 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:30.449468 2574 generic.go:356] "Generic (PLEG): container finished" podID="d8089437-f546-441e-b121-6e79878d2608" containerID="cf2969e21000e9780f10eea9d68d869842f5e96f9bd0a43e5b22fe83d49a2b98" exitCode=0 Oct 30 23:58:30.449638 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:30.449524 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tc87d" event={"ID":"d8089437-f546-441e-b121-6e79878d2608","Type":"ContainerDied","Data":"cf2969e21000e9780f10eea9d68d869842f5e96f9bd0a43e5b22fe83d49a2b98"} Oct 30 23:58:31.153947 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:31.153911 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:58:31.154361 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:31.154061 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4bx" podUID="50c15c32-e5cd-426c-9e7d-c123f8480653" Oct 30 23:58:31.154361 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:31.154122 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:58:31.154361 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:31.154216 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjqfw" podUID="0adc098d-57a4-490e-89b7-b1b271a1c087" Oct 30 23:58:31.453733 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:31.453658 2574 generic.go:356] "Generic (PLEG): container finished" podID="d8089437-f546-441e-b121-6e79878d2608" containerID="814815e3349f2a8643c4870bb191fdf48911ad2fe41f586127b1302fab55bf2a" exitCode=0 Oct 30 23:58:31.453733 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:31.453699 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tc87d" event={"ID":"d8089437-f546-441e-b121-6e79878d2608","Type":"ContainerDied","Data":"814815e3349f2a8643c4870bb191fdf48911ad2fe41f586127b1302fab55bf2a"} Oct 30 23:58:32.153980 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:32.153933 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:58:32.154345 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:32.154068 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6l8ht" podUID="99b4d92d-2003-4708-99da-a5f12cbb10e2" Oct 30 23:58:32.459179 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:32.459103 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tc87d" event={"ID":"d8089437-f546-441e-b121-6e79878d2608","Type":"ContainerStarted","Data":"ca6977ecca38c9c1dc4894b63829b7a2cb02bea1e526bab1c37ec5f8b1fdc3ed"} Oct 30 23:58:32.486518 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:32.486482 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tc87d" podStartSLOduration=7.690791883 podStartE2EDuration="19.486468841s" totalImagesPullingTime="11.795676958s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:58:13 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:32.486349916 +0000 UTC m=+19.803499232" watchObservedRunningTime="2026-10-30 23:58:32.486468841 +0000 UTC m=+19.803618150" Oct 30 23:58:33.155142 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:33.155113 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:58:33.155493 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:33.155222 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4bx" podUID="50c15c32-e5cd-426c-9e7d-c123f8480653" Oct 30 23:58:33.155493 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:33.155236 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:58:33.155493 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:33.155335 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjqfw" podUID="0adc098d-57a4-490e-89b7-b1b271a1c087" Oct 30 23:58:33.856268 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:33.856243 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6hq7b_d7815306-7a99-4fc5-8226-b34916ceaaa2/dns-node-resolver/0.log" Oct 30 23:58:34.154136 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:34.154065 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:58:34.154261 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:34.154192 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6l8ht" podUID="99b4d92d-2003-4708-99da-a5f12cbb10e2" Oct 30 23:58:35.154228 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:35.154199 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:58:35.154702 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:35.154364 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4bx" podUID="50c15c32-e5cd-426c-9e7d-c123f8480653" Oct 30 23:58:35.154702 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:35.154404 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:58:35.154702 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:35.154486 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjqfw" podUID="0adc098d-57a4-490e-89b7-b1b271a1c087" Oct 30 23:58:35.229999 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:35.229974 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vnk6j_18acd0a8-988e-4168-84e7-0b25dc7b32d4/node-ca/0.log" Oct 30 23:58:36.153887 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:36.153858 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:58:36.154056 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:36.153977 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6l8ht" podUID="99b4d92d-2003-4708-99da-a5f12cbb10e2" Oct 30 23:58:37.154068 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:37.154033 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:58:37.154068 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:37.154053 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:58:37.154632 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:37.154154 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4bx" podUID="50c15c32-e5cd-426c-9e7d-c123f8480653" Oct 30 23:58:37.154632 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:37.154277 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjqfw" podUID="0adc098d-57a4-490e-89b7-b1b271a1c087" Oct 30 23:58:38.154521 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:38.154489 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:58:38.154970 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:38.154638 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6l8ht" podUID="99b4d92d-2003-4708-99da-a5f12cbb10e2" Oct 30 23:58:39.154104 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:39.154074 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:58:39.154104 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:39.154091 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:58:39.154346 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:39.154198 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4bx" podUID="50c15c32-e5cd-426c-9e7d-c123f8480653" Oct 30 23:58:39.154346 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:39.154329 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjqfw" podUID="0adc098d-57a4-490e-89b7-b1b271a1c087" Oct 30 23:58:39.474135 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:39.474109 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" event={"ID":"25e99515-222f-4346-9fe5-5b7b3b213ee8","Type":"ContainerStarted","Data":"6d1b1d23ef41e86b4eddfb8ed832a972131ca2d2441e12ac515bfccff073dbde"} Oct 30 23:58:39.474465 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:39.474204 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" event={"ID":"25e99515-222f-4346-9fe5-5b7b3b213ee8","Type":"ContainerStarted","Data":"da4a197959c76c0256252d0edd04d7b492014bdbfaf3c59ff38748b7d0b7f9be"} Oct 30 23:58:39.474465 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:39.474234 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" event={"ID":"25e99515-222f-4346-9fe5-5b7b3b213ee8","Type":"ContainerStarted","Data":"e10f1d12f7699685766967ef0c95c4f38b2bb484498c098f7844e84944c10e8e"} Oct 30 23:58:40.154256 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:40.154226 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:58:40.154441 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:40.154333 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6l8ht" podUID="99b4d92d-2003-4708-99da-a5f12cbb10e2" Oct 30 23:58:40.479128 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:40.479057 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" event={"ID":"25e99515-222f-4346-9fe5-5b7b3b213ee8","Type":"ContainerStarted","Data":"108d63873eca5ee8435edd2e9cb22465afa6ceff0ca87e09deedec3258355e02"} Oct 30 23:58:40.479128 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:40.479121 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" event={"ID":"25e99515-222f-4346-9fe5-5b7b3b213ee8","Type":"ContainerStarted","Data":"0ef89aa8b0f3a39eef1b62ec0845315c6f60d05437c40028b39e86d2859df2c2"} Oct 30 23:58:40.479602 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:40.479156 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" event={"ID":"25e99515-222f-4346-9fe5-5b7b3b213ee8","Type":"ContainerStarted","Data":"29ebca814a19545edd8440c728191a51e694be4e17fbc065b68255688e920c61"} Oct 30 23:58:41.153793 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:41.153763 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:58:41.154056 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:41.153857 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4bx" podUID="50c15c32-e5cd-426c-9e7d-c123f8480653" Oct 30 23:58:41.154056 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:41.153975 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:58:41.154137 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:41.154065 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjqfw" podUID="0adc098d-57a4-490e-89b7-b1b271a1c087" Oct 30 23:58:42.154409 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:42.154390 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:58:42.154782 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:42.154490 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6l8ht" podUID="99b4d92d-2003-4708-99da-a5f12cbb10e2" Oct 30 23:58:42.486166 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:42.486086 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" event={"ID":"25e99515-222f-4346-9fe5-5b7b3b213ee8","Type":"ContainerStarted","Data":"7510758ac27233bdb79fda369344392a104d7d5be1f31d21ba0a3ff2400914c4"} Oct 30 23:58:42.822822 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:42.822761 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-x962k" Oct 30 23:58:42.822967 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:42.822938 2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 23:58:42.823416 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:42.823398 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-x962k" Oct 30 23:58:43.155238 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:43.155171 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:58:43.155585 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:43.155244 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4bx" podUID="50c15c32-e5cd-426c-9e7d-c123f8480653" Oct 30 23:58:43.155585 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:43.155349 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:58:43.155585 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:43.155463 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjqfw" podUID="0adc098d-57a4-490e-89b7-b1b271a1c087" Oct 30 23:58:44.153686 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:44.153593 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:58:44.153823 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:44.153689 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6l8ht" podUID="99b4d92d-2003-4708-99da-a5f12cbb10e2" Oct 30 23:58:44.494580 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:44.494504 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" event={"ID":"25e99515-222f-4346-9fe5-5b7b3b213ee8","Type":"ContainerStarted","Data":"ee78340c4a3d5f689eddee3c926c34ee97b9d48333258d6b8e39e08137c1b895"} Oct 30 23:58:44.495071 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:44.494709 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:44.495071 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:44.494767 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:44.495071 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:44.494817 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:44.510863 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:44.510843 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:44.510999 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:44.510937 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:58:44.529970 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:44.529896 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" podStartSLOduration=5.71814797 podStartE2EDuration="30.529881237s" totalImagesPullingTime="24.811733267s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:58:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:44.527922393 +0000 UTC m=+31.845071704" watchObservedRunningTime="2026-10-30 23:58:44.529881237 +0000 UTC m=+31.847030550" Oct 30 23:58:45.154024 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:45.153990 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:58:45.154265 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:45.154184 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4bx" podUID="50c15c32-e5cd-426c-9e7d-c123f8480653" Oct 30 23:58:45.154265 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:45.154220 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:58:45.154382 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:45.154299 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjqfw" podUID="0adc098d-57a4-490e-89b7-b1b271a1c087" Oct 30 23:58:45.886111 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:45.886078 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99b4d92d-2003-4708-99da-a5f12cbb10e2-original-pull-secret\") pod \"global-pull-secret-syncer-6l8ht\" (UID: \"99b4d92d-2003-4708-99da-a5f12cbb10e2\") " pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:58:45.886540 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:45.886134 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0adc098d-57a4-490e-89b7-b1b271a1c087-metrics-certs\") pod \"network-metrics-daemon-cjqfw\" (UID: \"0adc098d-57a4-490e-89b7-b1b271a1c087\") " pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:58:45.886540 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:45.886218 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:58:45.886540 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:45.886265 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0adc098d-57a4-490e-89b7-b1b271a1c087-metrics-certs podName:0adc098d-57a4-490e-89b7-b1b271a1c087 nodeName:}" failed. No retries permitted until 2026-10-30 23:59:17.88625194 +0000 UTC m=+65.203401243 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0adc098d-57a4-490e-89b7-b1b271a1c087-metrics-certs") pod "network-metrics-daemon-cjqfw" (UID: "0adc098d-57a4-490e-89b7-b1b271a1c087") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:58:45.886540 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:45.886218 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:58:45.886540 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:45.886390 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99b4d92d-2003-4708-99da-a5f12cbb10e2-original-pull-secret podName:99b4d92d-2003-4708-99da-a5f12cbb10e2 nodeName:}" failed. No retries permitted until 2026-10-30 23:59:17.88636574 +0000 UTC m=+65.203515051 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99b4d92d-2003-4708-99da-a5f12cbb10e2-original-pull-secret") pod "global-pull-secret-syncer-6l8ht" (UID: "99b4d92d-2003-4708-99da-a5f12cbb10e2") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:58:45.987018 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:45.986988 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd4j2\" (UniqueName: \"kubernetes.io/projected/50c15c32-e5cd-426c-9e7d-c123f8480653-kube-api-access-gd4j2\") pod \"network-check-target-bd4bx\" (UID: \"50c15c32-e5cd-426c-9e7d-c123f8480653\") " pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:58:45.987157 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:45.987103 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:58:45.987157 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:45.987115 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:58:45.987157 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:45.987123 2574 projected.go:196] Error preparing data for projected volume kube-api-access-gd4j2 for pod openshift-network-diagnostics/network-check-target-bd4bx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:58:45.987245 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:45.987164 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50c15c32-e5cd-426c-9e7d-c123f8480653-kube-api-access-gd4j2 podName:50c15c32-e5cd-426c-9e7d-c123f8480653 nodeName:}" failed. No retries permitted until 2026-10-30 23:59:17.987152123 +0000 UTC m=+65.304301427 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gd4j2" (UniqueName: "kubernetes.io/projected/50c15c32-e5cd-426c-9e7d-c123f8480653-kube-api-access-gd4j2") pod "network-check-target-bd4bx" (UID: "50c15c32-e5cd-426c-9e7d-c123f8480653") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:58:46.154642 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:46.154583 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:58:46.154766 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:46.154670 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6l8ht" podUID="99b4d92d-2003-4708-99da-a5f12cbb10e2" Oct 30 23:58:47.153615 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:47.153586 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:58:47.154111 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:47.153586 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:58:47.154111 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:47.153705 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4bx" podUID="50c15c32-e5cd-426c-9e7d-c123f8480653" Oct 30 23:58:47.154111 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:47.153778 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjqfw" podUID="0adc098d-57a4-490e-89b7-b1b271a1c087" Oct 30 23:58:48.154335 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:48.154309 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:58:48.154708 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:48.154397 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6l8ht" podUID="99b4d92d-2003-4708-99da-a5f12cbb10e2" Oct 30 23:58:49.154714 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:49.154681 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:58:49.155117 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:49.154805 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4bx" podUID="50c15c32-e5cd-426c-9e7d-c123f8480653" Oct 30 23:58:49.155117 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:49.154854 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:58:49.155117 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:49.154943 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjqfw" podUID="0adc098d-57a4-490e-89b7-b1b271a1c087" Oct 30 23:58:50.153970 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:50.153932 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:58:50.154108 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:50.154032 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6l8ht" podUID="99b4d92d-2003-4708-99da-a5f12cbb10e2" Oct 30 23:58:51.153637 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:51.153608 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:58:51.154027 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:51.153726 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4bx" podUID="50c15c32-e5cd-426c-9e7d-c123f8480653" Oct 30 23:58:51.154027 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:51.153797 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:58:51.154027 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:51.153882 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjqfw" podUID="0adc098d-57a4-490e-89b7-b1b271a1c087" Oct 30 23:58:52.153706 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:52.153679 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:58:52.154113 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:52.153774 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6l8ht" podUID="99b4d92d-2003-4708-99da-a5f12cbb10e2" Oct 30 23:58:53.155435 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:53.155401 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:58:53.155803 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:53.155506 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4bx" podUID="50c15c32-e5cd-426c-9e7d-c123f8480653" Oct 30 23:58:53.155803 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:53.155556 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:58:53.155803 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:53.155647 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjqfw" podUID="0adc098d-57a4-490e-89b7-b1b271a1c087" Oct 30 23:58:54.157142 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:54.157113 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:58:54.157779 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:54.157739 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6l8ht" podUID="99b4d92d-2003-4708-99da-a5f12cbb10e2" Oct 30 23:58:54.574411 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:54.574326 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6l8ht"] Oct 30 23:58:54.574568 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:54.574438 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:58:54.574568 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:54.574541 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6l8ht" podUID="99b4d92d-2003-4708-99da-a5f12cbb10e2" Oct 30 23:58:54.584533 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:54.583742 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bd4bx"] Oct 30 23:58:54.585244 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:54.585220 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:58:54.585389 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:54.585356 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4bx" podUID="50c15c32-e5cd-426c-9e7d-c123f8480653" Oct 30 23:58:54.585526 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:54.585502 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cjqfw"] Oct 30 23:58:54.585622 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:54.585607 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:58:54.585736 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:54.585716 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjqfw" podUID="0adc098d-57a4-490e-89b7-b1b271a1c087" Oct 30 23:58:56.154504 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:56.154469 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:58:56.154933 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:56.154538 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:58:56.154933 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:56.154569 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:58:56.154933 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:56.154661 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4bx" podUID="50c15c32-e5cd-426c-9e7d-c123f8480653" Oct 30 23:58:56.154933 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:56.154696 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6l8ht" podUID="99b4d92d-2003-4708-99da-a5f12cbb10e2" Oct 30 23:58:56.154933 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:56.154777 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjqfw" podUID="0adc098d-57a4-490e-89b7-b1b271a1c087" Oct 30 23:58:58.154682 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:58.154497 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:58:58.155120 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:58.154507 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:58:58.155120 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:58.154752 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6l8ht" podUID="99b4d92d-2003-4708-99da-a5f12cbb10e2" Oct 30 23:58:58.155120 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:58.154810 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bd4bx" podUID="50c15c32-e5cd-426c-9e7d-c123f8480653" Oct 30 23:58:58.155120 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:58.154544 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:58:58.155120 ip-10-0-129-70 kubenswrapper[2574]: E1030 23:58:58.154921 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjqfw" podUID="0adc098d-57a4-490e-89b7-b1b271a1c087" Oct 30 23:58:59.510461 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.510434 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-70.us-east-2.compute.internal" event="NodeReady" Oct 30 23:58:59.510823 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.510567 2574 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 23:58:59.574678 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.574645 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-84g8j"] Oct 30 23:58:59.577328 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.577311 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-djtcv"] Oct 30 23:58:59.577454 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.577437 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-84g8j" Oct 30 23:58:59.580118 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.580069 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-djtcv" Oct 30 23:58:59.589123 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.589100 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-djtcv"] Oct 30 23:58:59.591778 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.591549 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-84g8j"] Oct 30 23:58:59.594639 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.594617 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-g5p2k"] Oct 30 23:58:59.597563 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.597546 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-g5p2k" Oct 30 23:58:59.613888 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.613867 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-g5p2k"] Oct 30 23:58:59.655508 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.655480 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 23:58:59.655770 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.655709 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 23:58:59.655770 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.655729 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 23:58:59.655770 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.655747 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 23:58:59.655770 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.655738 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 23:58:59.655770 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.655770 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 23:58:59.655983 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.655737 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 23:58:59.655983 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.655787 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 23:58:59.655983 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.655746 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lxfk8\"" Oct 30 23:58:59.655983 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.655938 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 23:58:59.656570 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.656557 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-vmg6c\"" Oct 30 23:58:59.656613 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.656581 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l7b2l\"" Oct 30 23:58:59.685796 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.685775 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnlg7\" (UniqueName: \"kubernetes.io/projected/8b099593-ded8-475c-b5ef-a91bbd9aeb1a-kube-api-access-gnlg7\") pod \"dns-default-g5p2k\" (UID: \"8b099593-ded8-475c-b5ef-a91bbd9aeb1a\") " pod="openshift-dns/dns-default-g5p2k" Oct 30 23:58:59.685866 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.685821 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3cb65abb-ba0e-4075-9a0a-455e1b64b820-data-volume\") pod \"insights-runtime-extractor-84g8j\" (UID: \"3cb65abb-ba0e-4075-9a0a-455e1b64b820\") " pod="openshift-insights/insights-runtime-extractor-84g8j" Oct 30 23:58:59.685866 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.685841 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3cb65abb-ba0e-4075-9a0a-455e1b64b820-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-84g8j\" (UID: \"3cb65abb-ba0e-4075-9a0a-455e1b64b820\") " pod="openshift-insights/insights-runtime-extractor-84g8j" Oct 30 23:58:59.685934 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.685904 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqkgh\" (UniqueName: \"kubernetes.io/projected/18698607-3254-4ec9-b3bb-bbe59c6c744f-kube-api-access-cqkgh\") pod \"ingress-canary-djtcv\" (UID: \"18698607-3254-4ec9-b3bb-bbe59c6c744f\") " pod="openshift-ingress-canary/ingress-canary-djtcv" Oct 30 23:58:59.685986 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.685932 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b099593-ded8-475c-b5ef-a91bbd9aeb1a-tmp-dir\") pod \"dns-default-g5p2k\" (UID: \"8b099593-ded8-475c-b5ef-a91bbd9aeb1a\") " pod="openshift-dns/dns-default-g5p2k" Oct 30 23:58:59.685986 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.685968 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3cb65abb-ba0e-4075-9a0a-455e1b64b820-crio-socket\") pod \"insights-runtime-extractor-84g8j\" (UID: \"3cb65abb-ba0e-4075-9a0a-455e1b64b820\") " pod="openshift-insights/insights-runtime-extractor-84g8j" Oct 30 23:58:59.686051 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.685996 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsblw\" (UniqueName: \"kubernetes.io/projected/3cb65abb-ba0e-4075-9a0a-455e1b64b820-kube-api-access-wsblw\") pod \"insights-runtime-extractor-84g8j\" (UID: \"3cb65abb-ba0e-4075-9a0a-455e1b64b820\") " pod="openshift-insights/insights-runtime-extractor-84g8j" Oct 30 23:58:59.686082 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.686052 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b099593-ded8-475c-b5ef-a91bbd9aeb1a-metrics-tls\") pod \"dns-default-g5p2k\" (UID: \"8b099593-ded8-475c-b5ef-a91bbd9aeb1a\") " pod="openshift-dns/dns-default-g5p2k" Oct 30 23:58:59.686082 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.686077 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8b099593-ded8-475c-b5ef-a91bbd9aeb1a-config-volume\") pod \"dns-default-g5p2k\" (UID: \"8b099593-ded8-475c-b5ef-a91bbd9aeb1a\") " pod="openshift-dns/dns-default-g5p2k" Oct 30 23:58:59.686139 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.686116 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3cb65abb-ba0e-4075-9a0a-455e1b64b820-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-84g8j\" (UID: \"3cb65abb-ba0e-4075-9a0a-455e1b64b820\") " pod="openshift-insights/insights-runtime-extractor-84g8j" Oct 30 23:58:59.686172 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.686144 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18698607-3254-4ec9-b3bb-bbe59c6c744f-cert\") pod \"ingress-canary-djtcv\" (UID: \"18698607-3254-4ec9-b3bb-bbe59c6c744f\") " pod="openshift-ingress-canary/ingress-canary-djtcv" Oct 30 23:58:59.787158 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.787113 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnlg7\" (UniqueName: \"kubernetes.io/projected/8b099593-ded8-475c-b5ef-a91bbd9aeb1a-kube-api-access-gnlg7\") pod \"dns-default-g5p2k\" (UID: \"8b099593-ded8-475c-b5ef-a91bbd9aeb1a\") " pod="openshift-dns/dns-default-g5p2k" Oct 30 23:58:59.787261 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.787175 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3cb65abb-ba0e-4075-9a0a-455e1b64b820-data-volume\") pod \"insights-runtime-extractor-84g8j\" (UID: \"3cb65abb-ba0e-4075-9a0a-455e1b64b820\") " pod="openshift-insights/insights-runtime-extractor-84g8j" Oct 30 23:58:59.787261 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.787202 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3cb65abb-ba0e-4075-9a0a-455e1b64b820-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-84g8j\" (UID: \"3cb65abb-ba0e-4075-9a0a-455e1b64b820\") " pod="openshift-insights/insights-runtime-extractor-84g8j" Oct 30 23:58:59.787261 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.787222 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqkgh\" (UniqueName: \"kubernetes.io/projected/18698607-3254-4ec9-b3bb-bbe59c6c744f-kube-api-access-cqkgh\") pod \"ingress-canary-djtcv\" (UID: \"18698607-3254-4ec9-b3bb-bbe59c6c744f\") " pod="openshift-ingress-canary/ingress-canary-djtcv" Oct 30 23:58:59.787261 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.787237 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b099593-ded8-475c-b5ef-a91bbd9aeb1a-tmp-dir\") pod \"dns-default-g5p2k\" (UID: \"8b099593-ded8-475c-b5ef-a91bbd9aeb1a\") " pod="openshift-dns/dns-default-g5p2k" Oct 30 23:58:59.787261 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.787258 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3cb65abb-ba0e-4075-9a0a-455e1b64b820-crio-socket\") pod \"insights-runtime-extractor-84g8j\" (UID: \"3cb65abb-ba0e-4075-9a0a-455e1b64b820\") " pod="openshift-insights/insights-runtime-extractor-84g8j" Oct 30 23:58:59.787464 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.787281 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsblw\" (UniqueName: \"kubernetes.io/projected/3cb65abb-ba0e-4075-9a0a-455e1b64b820-kube-api-access-wsblw\") pod \"insights-runtime-extractor-84g8j\" (UID: \"3cb65abb-ba0e-4075-9a0a-455e1b64b820\") " pod="openshift-insights/insights-runtime-extractor-84g8j" Oct 30 23:58:59.787464 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.787316 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b099593-ded8-475c-b5ef-a91bbd9aeb1a-metrics-tls\") pod \"dns-default-g5p2k\" (UID: \"8b099593-ded8-475c-b5ef-a91bbd9aeb1a\") " pod="openshift-dns/dns-default-g5p2k" Oct 30 23:58:59.787464 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.787396 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3cb65abb-ba0e-4075-9a0a-455e1b64b820-crio-socket\") pod \"insights-runtime-extractor-84g8j\" (UID: \"3cb65abb-ba0e-4075-9a0a-455e1b64b820\") " pod="openshift-insights/insights-runtime-extractor-84g8j" Oct 30 23:58:59.787464 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.787447 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8b099593-ded8-475c-b5ef-a91bbd9aeb1a-config-volume\") pod \"dns-default-g5p2k\" (UID: \"8b099593-ded8-475c-b5ef-a91bbd9aeb1a\") " pod="openshift-dns/dns-default-g5p2k" Oct 30 23:58:59.787654 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.787491 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3cb65abb-ba0e-4075-9a0a-455e1b64b820-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-84g8j\" (UID: \"3cb65abb-ba0e-4075-9a0a-455e1b64b820\") " pod="openshift-insights/insights-runtime-extractor-84g8j" Oct 30 23:58:59.787654 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.787517 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18698607-3254-4ec9-b3bb-bbe59c6c744f-cert\") pod \"ingress-canary-djtcv\" (UID: \"18698607-3254-4ec9-b3bb-bbe59c6c744f\") " pod="openshift-ingress-canary/ingress-canary-djtcv" Oct 30 23:58:59.787654 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.787642 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3cb65abb-ba0e-4075-9a0a-455e1b64b820-data-volume\") pod \"insights-runtime-extractor-84g8j\" (UID: \"3cb65abb-ba0e-4075-9a0a-455e1b64b820\") " pod="openshift-insights/insights-runtime-extractor-84g8j" Oct 30 23:58:59.787794 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.787688 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b099593-ded8-475c-b5ef-a91bbd9aeb1a-tmp-dir\") pod \"dns-default-g5p2k\" (UID: \"8b099593-ded8-475c-b5ef-a91bbd9aeb1a\") " pod="openshift-dns/dns-default-g5p2k" Oct 30 23:58:59.788188 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.788164 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8b099593-ded8-475c-b5ef-a91bbd9aeb1a-config-volume\") pod \"dns-default-g5p2k\" (UID: \"8b099593-ded8-475c-b5ef-a91bbd9aeb1a\") " pod="openshift-dns/dns-default-g5p2k" Oct 30 23:58:59.788278 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.788164 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3cb65abb-ba0e-4075-9a0a-455e1b64b820-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-84g8j\" (UID: \"3cb65abb-ba0e-4075-9a0a-455e1b64b820\") " pod="openshift-insights/insights-runtime-extractor-84g8j" Oct 30 23:58:59.791230 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.791199 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3cb65abb-ba0e-4075-9a0a-455e1b64b820-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-84g8j\" (UID: \"3cb65abb-ba0e-4075-9a0a-455e1b64b820\") " pod="openshift-insights/insights-runtime-extractor-84g8j" Oct 30 23:58:59.791339 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.791236 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b099593-ded8-475c-b5ef-a91bbd9aeb1a-metrics-tls\") pod \"dns-default-g5p2k\" (UID: \"8b099593-ded8-475c-b5ef-a91bbd9aeb1a\") " pod="openshift-dns/dns-default-g5p2k" Oct 30 23:58:59.791397 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.791354 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18698607-3254-4ec9-b3bb-bbe59c6c744f-cert\") pod \"ingress-canary-djtcv\" (UID: \"18698607-3254-4ec9-b3bb-bbe59c6c744f\") " pod="openshift-ingress-canary/ingress-canary-djtcv" Oct 30 23:58:59.799928 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.799901 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqkgh\" (UniqueName: \"kubernetes.io/projected/18698607-3254-4ec9-b3bb-bbe59c6c744f-kube-api-access-cqkgh\") pod \"ingress-canary-djtcv\" (UID: \"18698607-3254-4ec9-b3bb-bbe59c6c744f\") " pod="openshift-ingress-canary/ingress-canary-djtcv" Oct 30 23:58:59.800208 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.800187 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsblw\" (UniqueName: \"kubernetes.io/projected/3cb65abb-ba0e-4075-9a0a-455e1b64b820-kube-api-access-wsblw\") pod \"insights-runtime-extractor-84g8j\" (UID: \"3cb65abb-ba0e-4075-9a0a-455e1b64b820\") " pod="openshift-insights/insights-runtime-extractor-84g8j" Oct 30 23:58:59.800430 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.800413 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnlg7\" (UniqueName: \"kubernetes.io/projected/8b099593-ded8-475c-b5ef-a91bbd9aeb1a-kube-api-access-gnlg7\") pod \"dns-default-g5p2k\" (UID: \"8b099593-ded8-475c-b5ef-a91bbd9aeb1a\") " pod="openshift-dns/dns-default-g5p2k" Oct 30 23:58:59.894641 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.894610 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-84g8j" Oct 30 23:58:59.897733 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.897713 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-djtcv" Oct 30 23:58:59.910164 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:58:59.910141 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-g5p2k" Oct 30 23:59:00.154286 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:00.154198 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:59:00.154455 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:00.154219 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:59:00.154455 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:00.154342 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:59:00.158440 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:00.158421 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 23:59:00.158712 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:00.158695 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-fl8mx\"" Oct 30 23:59:00.158794 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:00.158698 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xfgqn\"" Oct 30 23:59:00.159478 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:00.159461 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 23:59:00.159811 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:00.159790 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mmfft\"" Oct 30 23:59:00.159905 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:00.159858 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 23:59:00.159905 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:00.159866 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 23:59:00.190781 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:00.190756 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-djtcv"] Oct 30 23:59:00.194703 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:00.194680 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-84g8j"] Oct 30 23:59:00.194787 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:00.194732 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-g5p2k"] Oct 30 23:59:00.527118 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:00.527027 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-djtcv" event={"ID":"18698607-3254-4ec9-b3bb-bbe59c6c744f","Type":"ContainerStarted","Data":"409e18ce6451a89d480abb89b9d4c3cf0655defaeda3960aec5bffe99a10a7fc"} Oct 30 23:59:00.528117 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:00.528090 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g5p2k" event={"ID":"8b099593-ded8-475c-b5ef-a91bbd9aeb1a","Type":"ContainerStarted","Data":"bbef4aa45643805424f941e549c15f58ddc8ab57981152c689444be36ab768e7"} Oct 30 23:59:00.529360 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:00.529337 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-84g8j" event={"ID":"3cb65abb-ba0e-4075-9a0a-455e1b64b820","Type":"ContainerStarted","Data":"37a2cbdd8732710a14a3a7f593b7df41b8ac38cbb66529ced1942ed9ec48fb65"} Oct 30 23:59:00.529433 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:00.529380 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-84g8j" event={"ID":"3cb65abb-ba0e-4075-9a0a-455e1b64b820","Type":"ContainerStarted","Data":"b665f50eb4a136f42a87c03c99aec73c08df17ec99489f998035c4252173865f"} Oct 30 23:59:01.534438 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:01.534362 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-84g8j" event={"ID":"3cb65abb-ba0e-4075-9a0a-455e1b64b820","Type":"ContainerStarted","Data":"2b0dfefd79d2061fffa5387fde20fdd40152987c32a6826c5b4653f816e90eb0"} Oct 30 23:59:02.539531 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.539493 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g5p2k" event={"ID":"8b099593-ded8-475c-b5ef-a91bbd9aeb1a","Type":"ContainerStarted","Data":"8d1369c7797c64c473703c0b8e2d988c59c797c4dcf301e4e9f5f0d9988e8e86"} Oct 30 23:59:02.540014 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.539557 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g5p2k" event={"ID":"8b099593-ded8-475c-b5ef-a91bbd9aeb1a","Type":"ContainerStarted","Data":"67695a717a43c4659ad86d912e67ef9a28f59452615ab376b0986da06e242e0f"} Oct 30 23:59:02.540014 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.539675 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-g5p2k" Oct 30 23:59:02.562363 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.562299 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-g5p2k" podStartSLOduration=1.7773864910000001 podStartE2EDuration="3.562279978s" totalImagesPullingTime="1.784893487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:02.561717447 +0000 UTC m=+49.878866765" watchObservedRunningTime="2026-10-30 23:59:02.562279978 +0000 UTC m=+49.879429290" Oct 30 23:59:02.742159 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.742130 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-tt54l"] Oct 30 23:59:02.745444 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.745419 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tt54l" Oct 30 23:59:02.765448 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.765416 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-tt54l"] Oct 30 23:59:02.767783 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.767756 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 23:59:02.767783 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.767789 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 23:59:02.767981 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.767789 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 23:59:02.767981 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.767846 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 23:59:02.767981 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.767965 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 23:59:02.768081 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.768048 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-mt26w\"" Oct 30 23:59:02.783992 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.783950 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc"] Oct 30 23:59:02.787115 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.787095 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc" Oct 30 23:59:02.803817 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.803788 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bpczc"] Oct 30 23:59:02.809306 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.809280 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:02.815740 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.815712 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc"] Oct 30 23:59:02.839890 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.839859 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 23:59:02.840370 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.840165 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 23:59:02.840370 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.840254 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 23:59:02.840535 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.840383 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-splp4\"" Oct 30 23:59:02.840535 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.840456 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xcdsb\"" Oct 30 23:59:02.840535 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.840532 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 23:59:02.871392 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.871352 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 23:59:02.871392 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.871364 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 23:59:02.912201 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.912170 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-node-exporter-textfile\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:02.912359 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.912207 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1ad2242c-28d7-4495-9bac-f6c8cba7f796-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-frfhc\" (UID: \"1ad2242c-28d7-4495-9bac-f6c8cba7f796\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc" Oct 30 23:59:02.912359 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.912236 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-metrics-client-ca\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:02.912430 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.912357 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ab2f199-64ea-4071-b8cd-6b9861d31179-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-tt54l\" (UID: \"5ab2f199-64ea-4071-b8cd-6b9861d31179\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tt54l" Oct 30 23:59:02.912430 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.912399 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:02.912430 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.912418 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ab2f199-64ea-4071-b8cd-6b9861d31179-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-tt54l\" (UID: \"5ab2f199-64ea-4071-b8cd-6b9861d31179\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tt54l" Oct 30 23:59:02.912534 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.912443 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-root\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:02.912534 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.912463 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-node-exporter-wtmp\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:02.912534 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.912484 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ad2242c-28d7-4495-9bac-f6c8cba7f796-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-frfhc\" (UID: \"1ad2242c-28d7-4495-9bac-f6c8cba7f796\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc" Oct 30 23:59:02.912534 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.912526 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-sys\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:02.912684 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.912627 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1ad2242c-28d7-4495-9bac-f6c8cba7f796-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-frfhc\" (UID: \"1ad2242c-28d7-4495-9bac-f6c8cba7f796\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc" Oct 30 23:59:02.912734 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.912717 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-node-exporter-accelerators-collector-config\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:02.912820 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.912800 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfgxg\" (UniqueName: \"kubernetes.io/projected/5ab2f199-64ea-4071-b8cd-6b9861d31179-kube-api-access-qfgxg\") pod \"openshift-state-metrics-7bd7489957-tt54l\" (UID: \"5ab2f199-64ea-4071-b8cd-6b9861d31179\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tt54l" Oct 30 23:59:02.912878 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.912865 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ab2f199-64ea-4071-b8cd-6b9861d31179-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-tt54l\" (UID: \"5ab2f199-64ea-4071-b8cd-6b9861d31179\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tt54l" Oct 30 23:59:02.912920 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.912894 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ad2242c-28d7-4495-9bac-f6c8cba7f796-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-frfhc\" (UID: \"1ad2242c-28d7-4495-9bac-f6c8cba7f796\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc" Oct 30 23:59:02.912920 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.912912 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ad2242c-28d7-4495-9bac-f6c8cba7f796-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-frfhc\" (UID: \"1ad2242c-28d7-4495-9bac-f6c8cba7f796\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc" Oct 30 23:59:02.913041 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.912936 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85v2m\" (UniqueName: \"kubernetes.io/projected/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-kube-api-access-85v2m\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:02.913041 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.912992 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gprnc\" (UniqueName: \"kubernetes.io/projected/1ad2242c-28d7-4495-9bac-f6c8cba7f796-kube-api-access-gprnc\") pod \"kube-state-metrics-5dc8d7d89b-frfhc\" (UID: \"1ad2242c-28d7-4495-9bac-f6c8cba7f796\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc" Oct 30 23:59:02.913041 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:02.913028 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-node-exporter-tls\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:03.013711 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.013682 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ab2f199-64ea-4071-b8cd-6b9861d31179-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-tt54l\" (UID: \"5ab2f199-64ea-4071-b8cd-6b9861d31179\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tt54l" Oct 30 23:59:03.013806 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.013733 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:03.013806 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.013761 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ab2f199-64ea-4071-b8cd-6b9861d31179-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-tt54l\" (UID: \"5ab2f199-64ea-4071-b8cd-6b9861d31179\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tt54l" Oct 30 23:59:03.013806 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.013789 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-root\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:03.013967 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.013811 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-node-exporter-wtmp\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:03.013967 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.013835 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ad2242c-28d7-4495-9bac-f6c8cba7f796-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-frfhc\" (UID: \"1ad2242c-28d7-4495-9bac-f6c8cba7f796\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc" Oct 30 23:59:03.013967 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.013855 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-sys\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:03.013967 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.013883 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1ad2242c-28d7-4495-9bac-f6c8cba7f796-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-frfhc\" (UID: \"1ad2242c-28d7-4495-9bac-f6c8cba7f796\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc" Oct 30 23:59:03.013967 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.013887 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-root\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:03.013967 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.013910 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-node-exporter-accelerators-collector-config\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:03.013967 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.013946 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfgxg\" (UniqueName: \"kubernetes.io/projected/5ab2f199-64ea-4071-b8cd-6b9861d31179-kube-api-access-qfgxg\") pod \"openshift-state-metrics-7bd7489957-tt54l\" (UID: \"5ab2f199-64ea-4071-b8cd-6b9861d31179\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tt54l" Oct 30 23:59:03.014300 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.013991 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-sys\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:03.014300 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.014004 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ab2f199-64ea-4071-b8cd-6b9861d31179-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-tt54l\" (UID: \"5ab2f199-64ea-4071-b8cd-6b9861d31179\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tt54l" Oct 30 23:59:03.014300 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.014031 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ad2242c-28d7-4495-9bac-f6c8cba7f796-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-frfhc\" (UID: \"1ad2242c-28d7-4495-9bac-f6c8cba7f796\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc" Oct 30 23:59:03.014300 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.014055 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ad2242c-28d7-4495-9bac-f6c8cba7f796-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-frfhc\" (UID: \"1ad2242c-28d7-4495-9bac-f6c8cba7f796\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc" Oct 30 23:59:03.014300 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.014091 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85v2m\" (UniqueName: \"kubernetes.io/projected/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-kube-api-access-85v2m\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:03.014300 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.014121 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gprnc\" (UniqueName: \"kubernetes.io/projected/1ad2242c-28d7-4495-9bac-f6c8cba7f796-kube-api-access-gprnc\") pod \"kube-state-metrics-5dc8d7d89b-frfhc\" (UID: \"1ad2242c-28d7-4495-9bac-f6c8cba7f796\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc" Oct 30 23:59:03.014300 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.014147 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-node-exporter-tls\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:03.014300 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.014207 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-node-exporter-textfile\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:03.014300 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.014234 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1ad2242c-28d7-4495-9bac-f6c8cba7f796-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-frfhc\" (UID: \"1ad2242c-28d7-4495-9bac-f6c8cba7f796\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc" Oct 30 23:59:03.014300 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.014264 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-metrics-client-ca\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:03.014779 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.014647 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-node-exporter-accelerators-collector-config\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:03.015602 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.014852 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-metrics-client-ca\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:03.015602 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.014880 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1ad2242c-28d7-4495-9bac-f6c8cba7f796-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-frfhc\" (UID: \"1ad2242c-28d7-4495-9bac-f6c8cba7f796\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc" Oct 30 23:59:03.015602 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.015169 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-node-exporter-textfile\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:03.015602 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.015391 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1ad2242c-28d7-4495-9bac-f6c8cba7f796-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-frfhc\" (UID: \"1ad2242c-28d7-4495-9bac-f6c8cba7f796\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc" Oct 30 23:59:03.015602 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.015490 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-node-exporter-wtmp\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:03.015602 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.015552 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ab2f199-64ea-4071-b8cd-6b9861d31179-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-tt54l\" (UID: \"5ab2f199-64ea-4071-b8cd-6b9861d31179\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tt54l" Oct 30 23:59:03.015999 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.015927 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ad2242c-28d7-4495-9bac-f6c8cba7f796-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-frfhc\" (UID: \"1ad2242c-28d7-4495-9bac-f6c8cba7f796\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc" Oct 30 23:59:03.016662 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.016639 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ad2242c-28d7-4495-9bac-f6c8cba7f796-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-frfhc\" (UID: \"1ad2242c-28d7-4495-9bac-f6c8cba7f796\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc" Oct 30 23:59:03.017168 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.017145 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ab2f199-64ea-4071-b8cd-6b9861d31179-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-tt54l\" (UID: \"5ab2f199-64ea-4071-b8cd-6b9861d31179\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tt54l" Oct 30 23:59:03.017261 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.017221 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ab2f199-64ea-4071-b8cd-6b9861d31179-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-tt54l\" (UID: \"5ab2f199-64ea-4071-b8cd-6b9861d31179\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tt54l" Oct 30 23:59:03.017797 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.017775 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-node-exporter-tls\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:03.017933 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.017905 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ad2242c-28d7-4495-9bac-f6c8cba7f796-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-frfhc\" (UID: \"1ad2242c-28d7-4495-9bac-f6c8cba7f796\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc" Oct 30 23:59:03.018268 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.018251 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:03.031267 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.031202 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85v2m\" (UniqueName: \"kubernetes.io/projected/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-kube-api-access-85v2m\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:03.031386 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.031281 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfgxg\" (UniqueName: \"kubernetes.io/projected/5ab2f199-64ea-4071-b8cd-6b9861d31179-kube-api-access-qfgxg\") pod \"openshift-state-metrics-7bd7489957-tt54l\" (UID: \"5ab2f199-64ea-4071-b8cd-6b9861d31179\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tt54l" Oct 30 23:59:03.031386 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.031305 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gprnc\" (UniqueName: \"kubernetes.io/projected/1ad2242c-28d7-4495-9bac-f6c8cba7f796-kube-api-access-gprnc\") pod \"kube-state-metrics-5dc8d7d89b-frfhc\" (UID: \"1ad2242c-28d7-4495-9bac-f6c8cba7f796\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc" Oct 30 23:59:03.062677 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.062597 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tt54l" Oct 30 23:59:03.106889 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.106842 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc" Oct 30 23:59:03.132553 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.132520 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:03.345102 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.345077 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-tt54l"] Oct 30 23:59:03.347475 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:59:03.347449 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ab2f199_64ea_4071_b8cd_6b9861d31179.slice/crio-fe4d0060951c6651063102dcb213c97f5b3d2d38e0cc975985f5dedd64be8cf4 WatchSource:0}: Error finding container fe4d0060951c6651063102dcb213c97f5b3d2d38e0cc975985f5dedd64be8cf4: Status 404 returned error can't find the container with id fe4d0060951c6651063102dcb213c97f5b3d2d38e0cc975985f5dedd64be8cf4 Oct 30 23:59:03.398589 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.398564 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc"] Oct 30 23:59:03.401375 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:59:03.401346 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad2242c_28d7_4495_9bac_f6c8cba7f796.slice/crio-0a478aca183c6cfef37fda65497616cbc096d1e47640b0f003ca2605afba647e WatchSource:0}: Error finding container 0a478aca183c6cfef37fda65497616cbc096d1e47640b0f003ca2605afba647e: Status 404 returned error can't find the container with id 0a478aca183c6cfef37fda65497616cbc096d1e47640b0f003ca2605afba647e Oct 30 23:59:03.543663 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.543627 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc" event={"ID":"1ad2242c-28d7-4495-9bac-f6c8cba7f796","Type":"ContainerStarted","Data":"0a478aca183c6cfef37fda65497616cbc096d1e47640b0f003ca2605afba647e"} Oct 30 23:59:03.544795 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.544771 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bpczc" event={"ID":"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9","Type":"ContainerStarted","Data":"ced632c8d7bead7d8d4c32eb34db763bdb44e1c743c8ea92132b98980ca46cad"} Oct 30 23:59:03.546524 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.546498 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tt54l" event={"ID":"5ab2f199-64ea-4071-b8cd-6b9861d31179","Type":"ContainerStarted","Data":"32025cdeb63fe2e379599ce6653d16f5ba2caa82d44175638f214de4cfff714b"} Oct 30 23:59:03.546616 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.546544 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tt54l" event={"ID":"5ab2f199-64ea-4071-b8cd-6b9861d31179","Type":"ContainerStarted","Data":"18b37419b004ee69bf11263bb5b05eaaa6795305db4174be79d16f9c314a9baf"} Oct 30 23:59:03.546616 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.546568 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tt54l" event={"ID":"5ab2f199-64ea-4071-b8cd-6b9861d31179","Type":"ContainerStarted","Data":"fe4d0060951c6651063102dcb213c97f5b3d2d38e0cc975985f5dedd64be8cf4"} Oct 30 23:59:03.547857 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.547836 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-djtcv" event={"ID":"18698607-3254-4ec9-b3bb-bbe59c6c744f","Type":"ContainerStarted","Data":"793646dfe83a8e98daf6c877804ad0de653853071d3812b9f33d2e23eaf669e8"} Oct 30 23:59:03.549797 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.549774 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-84g8j" event={"ID":"3cb65abb-ba0e-4075-9a0a-455e1b64b820","Type":"ContainerStarted","Data":"786d866f8a62e81e514ff17217e93f0f2c73d30f485a7bc6b724348efea32bb0"} Oct 30 23:59:03.566706 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.566648 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-djtcv" podStartSLOduration=1.8116405009999998 podStartE2EDuration="4.566635589s" totalImagesPullingTime="2.754995088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:03.566145721 +0000 UTC m=+50.883295032" watchObservedRunningTime="2026-10-30 23:59:03.566635589 +0000 UTC m=+50.883784900" Oct 30 23:59:03.586569 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:03.586513 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-84g8j" podStartSLOduration=2.27553323 podStartE2EDuration="4.586500072s" totalImagesPullingTime="2.310966842s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:58:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:03.586174917 +0000 UTC m=+50.903324231" watchObservedRunningTime="2026-10-30 23:59:03.586500072 +0000 UTC m=+50.903649375" Oct 30 23:59:04.554925 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:04.554835 2574 generic.go:356] "Generic (PLEG): container finished" podID="de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9" containerID="a434bd5bf75f6333c98ef5bf7ba66491eeace7c6c464c27f7eba21c64661b912" exitCode=0 Oct 30 23:59:04.555378 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:04.555345 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bpczc" event={"ID":"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9","Type":"ContainerDied","Data":"a434bd5bf75f6333c98ef5bf7ba66491eeace7c6c464c27f7eba21c64661b912"} Oct 30 23:59:05.560247 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:05.560209 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc" event={"ID":"1ad2242c-28d7-4495-9bac-f6c8cba7f796","Type":"ContainerStarted","Data":"08f9317ab79661b4af45d984711deff05cf27836a3b34980630372a2500d1216"} Oct 30 23:59:05.560706 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:05.560274 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc" event={"ID":"1ad2242c-28d7-4495-9bac-f6c8cba7f796","Type":"ContainerStarted","Data":"667bb9933aafef42515b7a28b589a133e6653bcf26bbc30377038929e48167b1"} Oct 30 23:59:05.560706 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:05.560304 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc" event={"ID":"1ad2242c-28d7-4495-9bac-f6c8cba7f796","Type":"ContainerStarted","Data":"b71eda5804d8d8d4ca2163a7d75bcbe7eff3b94293c65d2c14a5615245171348"} Oct 30 23:59:05.562286 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:05.562262 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bpczc" event={"ID":"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9","Type":"ContainerStarted","Data":"9036ff0d06b40567022119f815a9c93bf2bf0a589f7a4e461db38b8d8ae35e3c"} Oct 30 23:59:05.562400 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:05.562307 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bpczc" event={"ID":"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9","Type":"ContainerStarted","Data":"3c32240501fed56cbf44ee6878b6ffd1e57add972ddbb9421b25764402734a7a"} Oct 30 23:59:05.564251 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:05.564222 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tt54l" event={"ID":"5ab2f199-64ea-4071-b8cd-6b9861d31179","Type":"ContainerStarted","Data":"a051640ddf1fa7389e91e7a31619ac2438eca9f6442df085b34b0c6c34c006a5"} Oct 30 23:59:05.582246 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:05.582199 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc" podStartSLOduration=2.03345394 podStartE2EDuration="3.582185995s" totalImagesPullingTime="1.548732055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:05.581292917 +0000 UTC m=+52.898442231" watchObservedRunningTime="2026-10-30 23:59:05.582185995 +0000 UTC m=+52.899335307" Oct 30 23:59:05.601657 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:05.601602 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bpczc" podStartSLOduration=2.479959329 podStartE2EDuration="3.601587192s" totalImagesPullingTime="1.042601582s" totalInitContainerRuntime="79.026281ms" isStatefulPod=true podCreationTimestamp="2026-10-30 23:59:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:05.601359419 +0000 UTC m=+52.918508729" watchObservedRunningTime="2026-10-30 23:59:05.601587192 +0000 UTC m=+52.918736505" Oct 30 23:59:05.621853 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:05.621793 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tt54l" podStartSLOduration=1.950831232 podStartE2EDuration="3.621777887s" totalImagesPullingTime="1.670946655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:05.621258864 +0000 UTC m=+52.938408176" watchObservedRunningTime="2026-10-30 23:59:05.621777887 +0000 UTC m=+52.938927199" Oct 30 23:59:07.338510 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:07.338473 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp"] Oct 30 23:59:07.341558 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:07.341544 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" Oct 30 23:59:07.347319 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:07.347294 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 23:59:07.347649 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:07.347597 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 23:59:07.352599 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:07.352575 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp"] Oct 30 23:59:07.354846 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:07.354824 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6fbfb6ff44-2sxgp\" (UID: \"0a7a184e-d0ef-4c2a-ab06-129e37b7e22e\") " pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" Oct 30 23:59:07.355032 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:07.354858 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-audit-log\") pod \"metrics-server-6fbfb6ff44-2sxgp\" (UID: \"0a7a184e-d0ef-4c2a-ab06-129e37b7e22e\") " pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" Oct 30 23:59:07.355032 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:07.354879 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fr52\" (UniqueName: \"kubernetes.io/projected/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-kube-api-access-5fr52\") pod \"metrics-server-6fbfb6ff44-2sxgp\" (UID: \"0a7a184e-d0ef-4c2a-ab06-129e37b7e22e\") " pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" Oct 30 23:59:07.355115 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:07.355084 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-secret-metrics-server-client-certs\") pod \"metrics-server-6fbfb6ff44-2sxgp\" (UID: \"0a7a184e-d0ef-4c2a-ab06-129e37b7e22e\") " pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" Oct 30 23:59:07.355150 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:07.355134 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-client-ca-bundle\") pod \"metrics-server-6fbfb6ff44-2sxgp\" (UID: \"0a7a184e-d0ef-4c2a-ab06-129e37b7e22e\") " pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" Oct 30 23:59:07.355193 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:07.355162 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-metrics-server-audit-profiles\") pod \"metrics-server-6fbfb6ff44-2sxgp\" (UID: \"0a7a184e-d0ef-4c2a-ab06-129e37b7e22e\") " pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" Oct 30 23:59:07.355272 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:07.355251 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-secret-metrics-server-tls\") pod \"metrics-server-6fbfb6ff44-2sxgp\" (UID: \"0a7a184e-d0ef-4c2a-ab06-129e37b7e22e\") " pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" Oct 30 23:59:07.358321 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:07.358302 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 23:59:07.358408 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:07.358344 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-x2vzx\"" Oct 30 23:59:07.358556 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:07.358545 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 23:59:07.359278 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:07.359262 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-b7saddsnpolru\"" Oct 30 23:59:07.455901 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:07.455878 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-secret-metrics-server-tls\") pod \"metrics-server-6fbfb6ff44-2sxgp\" (UID: \"0a7a184e-d0ef-4c2a-ab06-129e37b7e22e\") " pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" Oct 30 23:59:07.456047 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:07.455916 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6fbfb6ff44-2sxgp\" (UID: \"0a7a184e-d0ef-4c2a-ab06-129e37b7e22e\") " pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" Oct 30 23:59:07.456047 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:07.455946 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-audit-log\") pod \"metrics-server-6fbfb6ff44-2sxgp\" (UID: \"0a7a184e-d0ef-4c2a-ab06-129e37b7e22e\") " pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" Oct 30 23:59:07.456047 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:07.455989 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fr52\" (UniqueName: \"kubernetes.io/projected/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-kube-api-access-5fr52\") pod \"metrics-server-6fbfb6ff44-2sxgp\" (UID: \"0a7a184e-d0ef-4c2a-ab06-129e37b7e22e\") " pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" Oct 30 23:59:07.456047 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:07.456024 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-secret-metrics-server-client-certs\") pod \"metrics-server-6fbfb6ff44-2sxgp\" (UID: \"0a7a184e-d0ef-4c2a-ab06-129e37b7e22e\") " pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" Oct 30 23:59:07.456232 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:07.456062 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-client-ca-bundle\") pod \"metrics-server-6fbfb6ff44-2sxgp\" (UID: \"0a7a184e-d0ef-4c2a-ab06-129e37b7e22e\") " pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" Oct 30 23:59:07.456232 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:07.456096 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-metrics-server-audit-profiles\") pod \"metrics-server-6fbfb6ff44-2sxgp\" (UID: \"0a7a184e-d0ef-4c2a-ab06-129e37b7e22e\") " pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" Oct 30 23:59:07.456476 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:07.456453 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-audit-log\") pod \"metrics-server-6fbfb6ff44-2sxgp\" (UID: \"0a7a184e-d0ef-4c2a-ab06-129e37b7e22e\") " pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" Oct 30 23:59:07.456701 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:07.456663 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6fbfb6ff44-2sxgp\" (UID: \"0a7a184e-d0ef-4c2a-ab06-129e37b7e22e\") " pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" Oct 30 23:59:07.457040 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:07.457018 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-metrics-server-audit-profiles\") pod \"metrics-server-6fbfb6ff44-2sxgp\" (UID: \"0a7a184e-d0ef-4c2a-ab06-129e37b7e22e\") " pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" Oct 30 23:59:07.458669 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:07.458647 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-secret-metrics-server-tls\") pod \"metrics-server-6fbfb6ff44-2sxgp\" (UID: \"0a7a184e-d0ef-4c2a-ab06-129e37b7e22e\") " pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" Oct 30 23:59:07.458759 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:07.458742 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-client-ca-bundle\") pod \"metrics-server-6fbfb6ff44-2sxgp\" (UID: \"0a7a184e-d0ef-4c2a-ab06-129e37b7e22e\") " pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" Oct 30 23:59:07.458929 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:07.458909 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-secret-metrics-server-client-certs\") pod \"metrics-server-6fbfb6ff44-2sxgp\" (UID: \"0a7a184e-d0ef-4c2a-ab06-129e37b7e22e\") " pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" Oct 30 23:59:07.475840 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:07.475820 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fr52\" (UniqueName: \"kubernetes.io/projected/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-kube-api-access-5fr52\") pod \"metrics-server-6fbfb6ff44-2sxgp\" (UID: \"0a7a184e-d0ef-4c2a-ab06-129e37b7e22e\") " pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" Oct 30 23:59:07.652474 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:07.652450 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" Oct 30 23:59:07.927866 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:07.927659 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp"] Oct 30 23:59:07.930361 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:59:07.930332 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a7a184e_d0ef_4c2a_ab06_129e37b7e22e.slice/crio-50782c2accb92735384b1ec75130c51e641b2414a95d55743e69bc02be3519ca WatchSource:0}: Error finding container 50782c2accb92735384b1ec75130c51e641b2414a95d55743e69bc02be3519ca: Status 404 returned error can't find the container with id 50782c2accb92735384b1ec75130c51e641b2414a95d55743e69bc02be3519ca Oct 30 23:59:08.094496 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:08.094462 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5dc5757857-pp2xd"] Oct 30 23:59:08.098844 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:08.098818 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:08.111208 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:08.109906 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 23:59:08.113770 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:08.113744 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-plh7c\"" Oct 30 23:59:08.115982 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:08.115863 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 23:59:08.115982 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:08.115911 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 23:59:08.115982 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:08.115918 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 23:59:08.116209 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:08.116123 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 23:59:08.119470 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:08.119448 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5dc5757857-pp2xd"] Oct 30 23:59:08.120516 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:08.120497 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 23:59:08.161425 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:08.161404 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c8d0e1f0-620a-46c0-8234-628fb8acb572-federate-client-tls\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:08.161562 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:08.161431 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znq6p\" (UniqueName: \"kubernetes.io/projected/c8d0e1f0-620a-46c0-8234-628fb8acb572-kube-api-access-znq6p\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:08.161562 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:08.161452 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8d0e1f0-620a-46c0-8234-628fb8acb572-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:08.161562 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:08.161479 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8d0e1f0-620a-46c0-8234-628fb8acb572-serving-certs-ca-bundle\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:08.161562 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:08.161498 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c8d0e1f0-620a-46c0-8234-628fb8acb572-secret-telemeter-client\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:08.161767 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:08.161637 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c8d0e1f0-620a-46c0-8234-628fb8acb572-telemeter-client-tls\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:08.161767 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:08.161670 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8d0e1f0-620a-46c0-8234-628fb8acb572-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:08.161767 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:08.161751 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8d0e1f0-620a-46c0-8234-628fb8acb572-metrics-client-ca\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:08.262855 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:08.262793 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c8d0e1f0-620a-46c0-8234-628fb8acb572-federate-client-tls\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:08.262855 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:08.262829 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znq6p\" (UniqueName: \"kubernetes.io/projected/c8d0e1f0-620a-46c0-8234-628fb8acb572-kube-api-access-znq6p\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:08.263080 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:08.262856 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8d0e1f0-620a-46c0-8234-628fb8acb572-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:08.263080 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:08.262896 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8d0e1f0-620a-46c0-8234-628fb8acb572-serving-certs-ca-bundle\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:08.263080 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:08.262929 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c8d0e1f0-620a-46c0-8234-628fb8acb572-secret-telemeter-client\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:08.263236 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:08.263082 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c8d0e1f0-620a-46c0-8234-628fb8acb572-telemeter-client-tls\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:08.263236 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:08.263117 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8d0e1f0-620a-46c0-8234-628fb8acb572-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:08.263236 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:08.263195 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8d0e1f0-620a-46c0-8234-628fb8acb572-metrics-client-ca\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:08.263737 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:08.263705 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8d0e1f0-620a-46c0-8234-628fb8acb572-serving-certs-ca-bundle\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:08.263887 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:08.263861 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8d0e1f0-620a-46c0-8234-628fb8acb572-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:08.263982 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:08.263920 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8d0e1f0-620a-46c0-8234-628fb8acb572-metrics-client-ca\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:08.265427 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:08.265404 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c8d0e1f0-620a-46c0-8234-628fb8acb572-secret-telemeter-client\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:08.265516 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:08.265471 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c8d0e1f0-620a-46c0-8234-628fb8acb572-federate-client-tls\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:08.265722 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:08.265705 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8d0e1f0-620a-46c0-8234-628fb8acb572-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:08.265783 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:08.265768 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c8d0e1f0-620a-46c0-8234-628fb8acb572-telemeter-client-tls\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:08.276306 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:08.276285 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znq6p\" (UniqueName: \"kubernetes.io/projected/c8d0e1f0-620a-46c0-8234-628fb8acb572-kube-api-access-znq6p\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:08.424510 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:08.424482 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:08.576332 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:08.576259 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" event={"ID":"0a7a184e-d0ef-4c2a-ab06-129e37b7e22e","Type":"ContainerStarted","Data":"50782c2accb92735384b1ec75130c51e641b2414a95d55743e69bc02be3519ca"} Oct 30 23:59:08.735773 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:08.735716 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5dc5757857-pp2xd"] Oct 30 23:59:08.739355 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:59:08.739325 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8d0e1f0_620a_46c0_8234_628fb8acb572.slice/crio-cca062ee4fe9f407fdab48547ae139d0d1957b3acbbce1c68840c495f196fd4a WatchSource:0}: Error finding container cca062ee4fe9f407fdab48547ae139d0d1957b3acbbce1c68840c495f196fd4a: Status 404 returned error can't find the container with id cca062ee4fe9f407fdab48547ae139d0d1957b3acbbce1c68840c495f196fd4a Oct 30 23:59:09.580243 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:09.580207 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" event={"ID":"c8d0e1f0-620a-46c0-8234-628fb8acb572","Type":"ContainerStarted","Data":"cca062ee4fe9f407fdab48547ae139d0d1957b3acbbce1c68840c495f196fd4a"} Oct 30 23:59:10.585208 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:10.585171 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" event={"ID":"0a7a184e-d0ef-4c2a-ab06-129e37b7e22e","Type":"ContainerStarted","Data":"d82aa5bcda7d979017150b5f57414b720318bfc3b1520b8b07542d34432536fb"} Oct 30 23:59:10.607972 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:10.607904 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" podStartSLOduration=1.8036105519999999 podStartE2EDuration="3.607888397s" totalImagesPullingTime="1.804277845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:10.607271494 +0000 UTC m=+57.924420814" watchObservedRunningTime="2026-10-30 23:59:10.607888397 +0000 UTC m=+57.925037710" Oct 30 23:59:12.552750 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:12.552715 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-g5p2k" Oct 30 23:59:12.596241 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:12.596204 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" event={"ID":"c8d0e1f0-620a-46c0-8234-628fb8acb572","Type":"ContainerStarted","Data":"e2c6c051f9eab41c94b193caaf1c73d87ee4fa4492ecff4ee3d7a2662168be3e"} Oct 30 23:59:13.602080 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:13.602046 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" event={"ID":"c8d0e1f0-620a-46c0-8234-628fb8acb572","Type":"ContainerStarted","Data":"efc7aaa8dd523b3288c0a2966d7ef102a48de279ecf5a26aca1cbf8043e53cb0"} Oct 30 23:59:13.602454 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:13.602100 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" event={"ID":"c8d0e1f0-620a-46c0-8234-628fb8acb572","Type":"ContainerStarted","Data":"416d1d5b8fe21423dc3c3b9608d2ba782914ac8ae3658f05f3fc46a67ee8a5f2"} Oct 30 23:59:13.630704 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:13.630647 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" podStartSLOduration=1.704429519 podStartE2EDuration="5.630633065s" totalImagesPullingTime="3.926203546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:13.628511829 +0000 UTC m=+60.945661140" watchObservedRunningTime="2026-10-30 23:59:13.630633065 +0000 UTC m=+60.947782426" Oct 30 23:59:16.509672 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:16.509642 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:17.949222 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:17.949162 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99b4d92d-2003-4708-99da-a5f12cbb10e2-original-pull-secret\") pod \"global-pull-secret-syncer-6l8ht\" (UID: \"99b4d92d-2003-4708-99da-a5f12cbb10e2\") " pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:59:17.949718 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:17.949331 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0adc098d-57a4-490e-89b7-b1b271a1c087-metrics-certs\") pod \"network-metrics-daemon-cjqfw\" (UID: \"0adc098d-57a4-490e-89b7-b1b271a1c087\") " pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:59:17.953078 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:17.953056 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 23:59:17.953078 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:17.953056 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 23:59:17.962537 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:17.962507 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99b4d92d-2003-4708-99da-a5f12cbb10e2-original-pull-secret\") pod \"global-pull-secret-syncer-6l8ht\" (UID: \"99b4d92d-2003-4708-99da-a5f12cbb10e2\") " pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:59:17.962537 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:17.962535 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0adc098d-57a4-490e-89b7-b1b271a1c087-metrics-certs\") pod \"network-metrics-daemon-cjqfw\" (UID: \"0adc098d-57a4-490e-89b7-b1b271a1c087\") " pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:59:18.050225 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:18.050189 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd4j2\" (UniqueName: \"kubernetes.io/projected/50c15c32-e5cd-426c-9e7d-c123f8480653-kube-api-access-gd4j2\") pod \"network-check-target-bd4bx\" (UID: \"50c15c32-e5cd-426c-9e7d-c123f8480653\") " pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:59:18.052748 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:18.052727 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 23:59:18.063347 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:18.063321 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 23:59:18.073746 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:18.073722 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd4j2\" (UniqueName: \"kubernetes.io/projected/50c15c32-e5cd-426c-9e7d-c123f8480653-kube-api-access-gd4j2\") pod \"network-check-target-bd4bx\" (UID: \"50c15c32-e5cd-426c-9e7d-c123f8480653\") " pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:59:18.169574 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:18.169541 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mmfft\"" Oct 30 23:59:18.177027 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:18.177005 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:59:18.177337 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:18.177321 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-fl8mx\"" Oct 30 23:59:18.184673 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:18.184649 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xfgqn\"" Oct 30 23:59:18.185749 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:18.185729 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:59:18.192826 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:18.192802 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:59:18.454473 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:18.454398 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6l8ht"] Oct 30 23:59:18.457261 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:59:18.457231 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99b4d92d_2003_4708_99da_a5f12cbb10e2.slice/crio-41dc65a16bae60af4f6fc3d44fe15f521d3a3580053e347ceb1ee9f5339e1749 WatchSource:0}: Error finding container 41dc65a16bae60af4f6fc3d44fe15f521d3a3580053e347ceb1ee9f5339e1749: Status 404 returned error can't find the container with id 41dc65a16bae60af4f6fc3d44fe15f521d3a3580053e347ceb1ee9f5339e1749 Oct 30 23:59:18.471507 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:18.471457 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bd4bx"] Oct 30 23:59:18.474199 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:59:18.474169 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50c15c32_e5cd_426c_9e7d_c123f8480653.slice/crio-707fc65a7121853641a6a4818f86b0ff658108656a1fe93eae4ab2f9b5e8fea1 WatchSource:0}: Error finding container 707fc65a7121853641a6a4818f86b0ff658108656a1fe93eae4ab2f9b5e8fea1: Status 404 returned error can't find the container with id 707fc65a7121853641a6a4818f86b0ff658108656a1fe93eae4ab2f9b5e8fea1 Oct 30 23:59:18.489447 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:18.489422 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cjqfw"] Oct 30 23:59:18.492187 ip-10-0-129-70 kubenswrapper[2574]: W1030 23:59:18.492161 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0adc098d_57a4_490e_89b7_b1b271a1c087.slice/crio-397510d0f15462f0e47d89b988695a3d37f74841306435abd01856455179c4c5 WatchSource:0}: Error finding container 397510d0f15462f0e47d89b988695a3d37f74841306435abd01856455179c4c5: Status 404 returned error can't find the container with id 397510d0f15462f0e47d89b988695a3d37f74841306435abd01856455179c4c5 Oct 30 23:59:18.621061 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:18.621019 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cjqfw" event={"ID":"0adc098d-57a4-490e-89b7-b1b271a1c087","Type":"ContainerStarted","Data":"397510d0f15462f0e47d89b988695a3d37f74841306435abd01856455179c4c5"} Oct 30 23:59:18.622202 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:18.622178 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bd4bx" event={"ID":"50c15c32-e5cd-426c-9e7d-c123f8480653","Type":"ContainerStarted","Data":"707fc65a7121853641a6a4818f86b0ff658108656a1fe93eae4ab2f9b5e8fea1"} Oct 30 23:59:18.623230 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:18.623207 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6l8ht" event={"ID":"99b4d92d-2003-4708-99da-a5f12cbb10e2","Type":"ContainerStarted","Data":"41dc65a16bae60af4f6fc3d44fe15f521d3a3580053e347ceb1ee9f5339e1749"} Oct 30 23:59:20.635122 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:20.635074 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cjqfw" event={"ID":"0adc098d-57a4-490e-89b7-b1b271a1c087","Type":"ContainerStarted","Data":"794c1b2f68635d535e03d35c1f625c6cc7620d4c11b0f7c87311c72b5faf2ff4"} Oct 30 23:59:20.635628 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:20.635145 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cjqfw" event={"ID":"0adc098d-57a4-490e-89b7-b1b271a1c087","Type":"ContainerStarted","Data":"8a7f48e3595e3aa7fb91f20f95a7193adcd62b3534a6354e72a6c29735773d9b"} Oct 30 23:59:20.656447 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:20.655546 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-cjqfw" podStartSLOduration=65.305348207 podStartE2EDuration="1m6.655528785s" totalImagesPullingTime="1.350180578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:20.655061542 +0000 UTC m=+67.972210857" watchObservedRunningTime="2026-10-30 23:59:20.655528785 +0000 UTC m=+67.972678133" Oct 30 23:59:22.643635 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:22.643591 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bd4bx" event={"ID":"50c15c32-e5cd-426c-9e7d-c123f8480653","Type":"ContainerStarted","Data":"45e4efec44b9553ed055a9c35e0066bd8f70a7697a2de190b7bdc13420e9376f"} Oct 30 23:59:22.644100 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:22.643757 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:59:22.664266 ip-10-0-129-70 kubenswrapper[2574]: I1030 23:59:22.664222 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bd4bx" podStartSLOduration=65.487896868 podStartE2EDuration="1m8.664208085s" totalImagesPullingTime="3.176311217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:22.662837532 +0000 UTC m=+69.979986845" watchObservedRunningTime="2026-10-30 23:59:22.664208085 +0000 UTC m=+69.981357398" Oct 30 23:59:23.301348 ip-10-0-129-70 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 23:59:23.315399 ip-10-0-129-70 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 23:59:23.315579 ip-10-0-129-70 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 23:59:23.315769 ip-10-0-129-70 systemd[1]: kubelet.service: Consumed 3.586s CPU time. Oct 30 23:59:23.328650 ip-10-0-129-70 systemd[1]: Starting Kubernetes Kubelet... Oct 30 23:59:23.394566 ip-10-0-129-70 kubenswrapper[5274]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:59:23.394566 ip-10-0-129-70 kubenswrapper[5274]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 23:59:23.394566 ip-10-0-129-70 kubenswrapper[5274]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:59:23.394566 ip-10-0-129-70 kubenswrapper[5274]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:59:23.396672 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396580 5274 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:59:23.396672 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396672 5274 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:59:23.396672 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396676 5274 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:59:23.396772 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396679 5274 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:59:23.396772 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396682 5274 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:59:23.396772 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396685 5274 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:59:23.396772 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396687 5274 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:59:23.396772 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396690 5274 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:59:23.396772 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396693 5274 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:59:23.396772 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396695 5274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:59:23.396772 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396699 5274 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:59:23.396772 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396702 5274 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:59:23.396772 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396705 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:59:23.396772 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396708 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:59:23.396772 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396710 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:59:23.396772 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396713 5274 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:59:23.396772 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396716 5274 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:59:23.396772 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396719 5274 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:59:23.396772 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396728 5274 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:59:23.396772 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396731 5274 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:59:23.396772 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396734 5274 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:59:23.396772 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396736 5274 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:59:23.396772 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396739 5274 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:59:23.397274 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396742 5274 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:59:23.397274 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396744 5274 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:59:23.397274 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396747 5274 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:59:23.397274 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396760 5274 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:59:23.397274 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396763 5274 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:59:23.397274 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396766 5274 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:59:23.397274 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396769 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:59:23.397274 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396771 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:59:23.397274 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396774 5274 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:59:23.397274 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396777 5274 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:59:23.397274 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396779 5274 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:59:23.397274 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396783 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:59:23.397274 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396785 5274 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:59:23.397274 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396788 5274 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:59:23.397274 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396791 5274 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:59:23.397274 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396793 5274 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:59:23.397274 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396796 5274 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:59:23.397274 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396799 5274 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:59:23.397274 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396801 5274 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:59:23.397734 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396804 5274 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:59:23.397734 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396807 5274 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:59:23.397734 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396809 5274 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:59:23.397734 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396812 5274 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:59:23.397734 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396814 5274 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:59:23.397734 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396818 5274 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:59:23.397734 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396820 5274 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:59:23.397734 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396823 5274 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:59:23.397734 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396826 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:59:23.397734 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396829 5274 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:59:23.397734 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396831 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:59:23.397734 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396835 5274 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:59:23.397734 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396837 5274 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:59:23.397734 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396840 5274 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:59:23.397734 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396843 5274 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:59:23.397734 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396845 5274 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:59:23.397734 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396848 5274 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:59:23.397734 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396856 5274 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:59:23.397734 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396859 5274 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:59:23.397734 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396862 5274 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:59:23.398285 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396864 5274 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:59:23.398285 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396867 5274 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:59:23.398285 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396870 5274 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:59:23.398285 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396873 5274 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:59:23.398285 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396875 5274 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:59:23.398285 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396878 5274 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:59:23.398285 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396880 5274 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:59:23.398285 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396883 5274 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:59:23.398285 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396885 5274 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:59:23.398285 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396888 5274 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:59:23.398285 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396891 5274 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:59:23.398285 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396896 5274 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:59:23.398285 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396900 5274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:59:23.398285 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396905 5274 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:59:23.398285 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396908 5274 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:59:23.398285 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396912 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:59:23.398285 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396914 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:59:23.398285 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396917 5274 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:59:23.398285 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396920 5274 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:59:23.398757 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396922 5274 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:59:23.398757 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396925 5274 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:59:23.398757 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396928 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:59:23.398757 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396931 5274 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:59:23.398757 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396933 5274 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:59:23.398757 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396936 5274 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:59:23.398757 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396938 5274 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:59:23.398757 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396941 5274 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:59:23.398757 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396944 5274 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:59:23.398757 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396946 5274 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:59:23.398757 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396949 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:59:23.398757 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396973 5274 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:59:23.398757 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396976 5274 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:59:23.398757 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396979 5274 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:59:23.398757 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396982 5274 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:59:23.398757 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396985 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:59:23.398757 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396987 5274 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:59:23.398757 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396990 5274 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:59:23.398757 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396993 5274 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:59:23.398757 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396996 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:59:23.399252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.396999 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:59:23.399252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397001 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:59:23.399252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397006 5274 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:59:23.399252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397008 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:59:23.399252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397011 5274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:59:23.399252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397806 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:59:23.399252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397812 5274 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:59:23.399252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397815 5274 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:59:23.399252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397818 5274 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:59:23.399252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397821 5274 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:59:23.399252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397824 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:59:23.399252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397827 5274 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:59:23.399252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397829 5274 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:59:23.399252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397833 5274 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:59:23.399252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397835 5274 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:59:23.399252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397838 5274 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:59:23.399252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397841 5274 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:59:23.399252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397844 5274 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:59:23.399252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397846 5274 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:59:23.399252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397849 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:59:23.399750 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397852 5274 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:59:23.399750 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397855 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:59:23.399750 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397857 5274 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:59:23.399750 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397860 5274 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:59:23.399750 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397868 5274 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:59:23.399750 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397871 5274 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:59:23.399750 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397874 5274 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:59:23.399750 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397876 5274 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:59:23.399750 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397879 5274 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:59:23.399750 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397882 5274 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:59:23.399750 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397884 5274 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:59:23.399750 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397888 5274 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:59:23.399750 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397890 5274 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:59:23.399750 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397893 5274 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:59:23.399750 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397896 5274 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:59:23.399750 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397899 5274 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:59:23.399750 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397901 5274 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:59:23.399750 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397904 5274 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:59:23.399750 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397906 5274 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:59:23.399750 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397909 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:59:23.400248 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397911 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:59:23.400248 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397914 5274 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:59:23.400248 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397916 5274 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:59:23.400248 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397921 5274 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:59:23.400248 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397926 5274 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:59:23.400248 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397929 5274 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:59:23.400248 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397932 5274 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:59:23.400248 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397935 5274 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:59:23.400248 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397938 5274 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:59:23.400248 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397940 5274 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:59:23.400248 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397943 5274 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:59:23.400248 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397945 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:59:23.400248 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397948 5274 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:59:23.400248 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397966 5274 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:59:23.400248 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397969 5274 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:59:23.400248 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397972 5274 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:59:23.400248 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397975 5274 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:59:23.400248 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397991 5274 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:59:23.400248 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397995 5274 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:59:23.400724 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.397998 5274 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:59:23.400724 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398000 5274 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:59:23.400724 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398003 5274 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:59:23.400724 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398006 5274 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:59:23.400724 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398008 5274 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:59:23.400724 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398011 5274 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:59:23.400724 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398013 5274 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:59:23.400724 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398017 5274 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:59:23.400724 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398020 5274 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:59:23.400724 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398023 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:59:23.400724 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398026 5274 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:59:23.400724 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398029 5274 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:59:23.400724 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398031 5274 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:59:23.400724 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398034 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:59:23.400724 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398036 5274 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:59:23.400724 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398039 5274 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:59:23.400724 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398041 5274 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:59:23.400724 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398045 5274 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:59:23.400724 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398047 5274 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:59:23.400724 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398050 5274 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:59:23.401218 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398052 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:59:23.401218 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398055 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:59:23.401218 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398058 5274 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:59:23.401218 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398061 5274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:59:23.401218 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398063 5274 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:59:23.401218 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398066 5274 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:59:23.401218 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398068 5274 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:59:23.401218 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398071 5274 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:59:23.401218 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398074 5274 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:59:23.401218 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398076 5274 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:59:23.401218 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398079 5274 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:59:23.401218 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398083 5274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:59:23.401218 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398085 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:59:23.401218 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398088 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:59:23.401218 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398091 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:59:23.401218 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398094 5274 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:59:23.401218 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398096 5274 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:59:23.401218 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398099 5274 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:59:23.401218 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398101 5274 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:59:23.401218 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398104 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:59:23.401701 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398106 5274 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:59:23.401701 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398109 5274 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:59:23.401701 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398111 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:59:23.401701 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398114 5274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:59:23.401701 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398117 5274 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:59:23.401701 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398119 5274 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:59:23.401701 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398122 5274 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:59:23.401701 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398124 5274 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:59:23.401701 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398126 5274 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:59:23.401701 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398129 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:59:23.401701 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398132 5274 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:59:23.401701 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398135 5274 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:59:23.401701 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398655 5274 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:59:23.401701 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398660 5274 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:59:23.401701 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398663 5274 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:59:23.401701 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398666 5274 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:59:23.401701 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398669 5274 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:59:23.401701 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398672 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:59:23.401701 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398675 5274 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:59:23.402178 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398677 5274 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:59:23.402178 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398680 5274 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:59:23.402178 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398683 5274 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:59:23.402178 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398687 5274 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:59:23.402178 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398692 5274 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:59:23.402178 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398695 5274 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:59:23.402178 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398698 5274 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:59:23.402178 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398701 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:59:23.402178 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398704 5274 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:59:23.402178 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398706 5274 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:59:23.402178 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398710 5274 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:59:23.402178 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398713 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:59:23.402178 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398716 5274 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:59:23.402178 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398718 5274 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:59:23.402178 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398721 5274 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:59:23.402178 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398723 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:59:23.402178 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398726 5274 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:59:23.402178 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398728 5274 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:59:23.402178 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398731 5274 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:59:23.402178 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398734 5274 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:59:23.402696 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398736 5274 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:59:23.402696 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398739 5274 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:59:23.402696 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398742 5274 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:59:23.402696 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398745 5274 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:59:23.402696 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398747 5274 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:59:23.402696 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398750 5274 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:59:23.402696 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398752 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:59:23.402696 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398755 5274 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:59:23.402696 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398758 5274 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:59:23.402696 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398761 5274 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:59:23.402696 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398764 5274 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:59:23.402696 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398767 5274 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:59:23.402696 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398771 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:59:23.402696 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398774 5274 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:59:23.402696 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398776 5274 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:59:23.402696 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398779 5274 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:59:23.402696 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398781 5274 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:59:23.402696 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398784 5274 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:59:23.402696 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398789 5274 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:59:23.402696 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398792 5274 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:59:23.403199 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398795 5274 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:59:23.403199 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398797 5274 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:59:23.403199 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398800 5274 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:59:23.403199 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398803 5274 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:59:23.403199 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398805 5274 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:59:23.403199 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398808 5274 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:59:23.403199 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398810 5274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:59:23.403199 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398813 5274 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:59:23.403199 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398815 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:59:23.403199 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398818 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:59:23.403199 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398821 5274 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:59:23.403199 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398823 5274 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:59:23.403199 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398826 5274 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:59:23.403199 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398829 5274 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:59:23.403199 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398832 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:59:23.403199 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398834 5274 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:59:23.403199 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398837 5274 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:59:23.403199 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398839 5274 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:59:23.403199 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398842 5274 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:59:23.403649 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398845 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:59:23.403649 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398851 5274 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:59:23.403649 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398854 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:59:23.403649 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398857 5274 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:59:23.403649 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398859 5274 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:59:23.403649 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398862 5274 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:59:23.403649 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398864 5274 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:59:23.403649 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398867 5274 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:59:23.403649 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398870 5274 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:59:23.403649 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398872 5274 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:59:23.403649 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398875 5274 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:59:23.403649 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398878 5274 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:59:23.403649 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398881 5274 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:59:23.403649 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398883 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:59:23.403649 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398886 5274 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:59:23.403649 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398888 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:59:23.403649 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398891 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:59:23.403649 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398894 5274 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:59:23.403649 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398896 5274 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:59:23.404156 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398899 5274 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:59:23.404156 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398901 5274 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:59:23.404156 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398904 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:59:23.404156 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398906 5274 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:59:23.404156 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398909 5274 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:59:23.404156 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398911 5274 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:59:23.404156 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398914 5274 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:59:23.404156 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398917 5274 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:59:23.404156 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398919 5274 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:59:23.404156 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398922 5274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:59:23.404156 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398925 5274 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:59:23.404156 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398928 5274 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:59:23.404156 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398931 5274 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:59:23.404156 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398934 5274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:59:23.404156 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398938 5274 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:59:23.404156 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398940 5274 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:59:23.404156 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398943 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:59:23.404156 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398946 5274 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:59:23.404156 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398948 5274 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:59:23.404156 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398965 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:59:23.404885 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.398970 5274 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:59:23.404885 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399083 5274 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 23:59:23.404885 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399093 5274 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 23:59:23.404885 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399099 5274 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 23:59:23.404885 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399104 5274 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 23:59:23.404885 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399109 5274 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 23:59:23.404885 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399113 5274 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 23:59:23.404885 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399118 5274 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 23:59:23.404885 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399123 5274 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 23:59:23.404885 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399126 5274 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 23:59:23.404885 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399130 5274 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 23:59:23.404885 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399133 5274 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 23:59:23.404885 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399137 5274 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 23:59:23.404885 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399141 5274 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 23:59:23.404885 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399144 5274 flags.go:64] FLAG: --cgroup-root="" Oct 30 23:59:23.404885 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399147 5274 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 23:59:23.404885 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399150 5274 flags.go:64] FLAG: --client-ca-file="" Oct 30 23:59:23.404885 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399155 5274 flags.go:64] FLAG: --cloud-provider="external" Oct 30 23:59:23.404885 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399158 5274 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 23:59:23.404885 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399163 5274 flags.go:64] FLAG: --cluster-domain="" Oct 30 23:59:23.404885 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399166 5274 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 23:59:23.404885 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399169 5274 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 23:59:23.404885 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399172 5274 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 23:59:23.404885 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399176 5274 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 23:59:23.405850 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399180 5274 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 23:59:23.405850 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399183 5274 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 23:59:23.405850 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399189 5274 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 23:59:23.405850 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399192 5274 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 23:59:23.405850 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399195 5274 flags.go:64] FLAG: --contention-profiling="false" Oct 30 23:59:23.405850 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399198 5274 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 23:59:23.405850 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399202 5274 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 23:59:23.405850 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399205 5274 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 23:59:23.405850 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399208 5274 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 23:59:23.405850 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399213 5274 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 23:59:23.405850 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399216 5274 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 23:59:23.405850 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399219 5274 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 23:59:23.405850 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399222 5274 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 23:59:23.405850 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399225 5274 flags.go:64] FLAG: --enable-server="true" Oct 30 23:59:23.405850 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399228 5274 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 23:59:23.405850 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399233 5274 flags.go:64] FLAG: --event-burst="100" Oct 30 23:59:23.405850 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399236 5274 flags.go:64] FLAG: --event-qps="50" Oct 30 23:59:23.405850 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399239 5274 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 23:59:23.405850 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399242 5274 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 23:59:23.405850 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399245 5274 flags.go:64] FLAG: --eviction-hard="" Oct 30 23:59:23.405850 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399249 5274 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 23:59:23.405850 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399252 5274 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 23:59:23.405850 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399256 5274 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 23:59:23.405850 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399259 5274 flags.go:64] FLAG: --eviction-soft="" Oct 30 23:59:23.405850 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399262 5274 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 23:59:23.406556 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399267 5274 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 23:59:23.406556 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399270 5274 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 23:59:23.406556 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399273 5274 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 23:59:23.406556 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399276 5274 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 23:59:23.406556 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399279 5274 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 23:59:23.406556 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399282 5274 flags.go:64] FLAG: --feature-gates="" Oct 30 23:59:23.406556 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399286 5274 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 23:59:23.406556 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399289 5274 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 23:59:23.406556 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399293 5274 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 23:59:23.406556 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399298 5274 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 23:59:23.406556 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399301 5274 flags.go:64] FLAG: --healthz-port="10248" Oct 30 23:59:23.406556 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399304 5274 flags.go:64] FLAG: --help="false" Oct 30 23:59:23.406556 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399308 5274 flags.go:64] FLAG: --hostname-override="ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:59:23.406556 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399311 5274 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 23:59:23.406556 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399314 5274 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 23:59:23.406556 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399317 5274 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 23:59:23.406556 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399321 5274 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 23:59:23.406556 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399324 5274 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 23:59:23.406556 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399327 5274 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 23:59:23.406556 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399330 5274 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 23:59:23.406556 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399334 5274 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 23:59:23.406556 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399337 5274 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 23:59:23.406556 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399340 5274 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 23:59:23.407140 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399343 5274 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 23:59:23.407140 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399346 5274 flags.go:64] FLAG: --kube-reserved="" Oct 30 23:59:23.407140 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399349 5274 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 23:59:23.407140 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399352 5274 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 23:59:23.407140 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399356 5274 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 23:59:23.407140 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399358 5274 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 23:59:23.407140 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399361 5274 flags.go:64] FLAG: --lock-file="" Oct 30 23:59:23.407140 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399364 5274 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 23:59:23.407140 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399367 5274 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 23:59:23.407140 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399375 5274 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 23:59:23.407140 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399382 5274 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 23:59:23.407140 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399385 5274 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 23:59:23.407140 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399388 5274 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 23:59:23.407140 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399392 5274 flags.go:64] FLAG: --logging-format="text" Oct 30 23:59:23.407140 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399395 5274 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 23:59:23.407140 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399398 5274 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 23:59:23.407140 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399401 5274 flags.go:64] FLAG: --manifest-url="" Oct 30 23:59:23.407140 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399404 5274 flags.go:64] FLAG: --manifest-url-header="" Oct 30 23:59:23.407140 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399410 5274 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 23:59:23.407140 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399413 5274 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 23:59:23.407140 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399417 5274 flags.go:64] FLAG: --max-pods="110" Oct 30 23:59:23.407140 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399421 5274 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 23:59:23.407140 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399424 5274 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 23:59:23.407140 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399427 5274 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 23:59:23.407140 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399430 5274 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 23:59:23.407746 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399433 5274 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 23:59:23.407746 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399436 5274 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 23:59:23.407746 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399439 5274 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 23:59:23.407746 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399446 5274 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 23:59:23.407746 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399450 5274 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 23:59:23.407746 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399453 5274 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 23:59:23.407746 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399456 5274 flags.go:64] FLAG: --pod-cidr="" Oct 30 23:59:23.407746 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399459 5274 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 23:59:23.407746 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399462 5274 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 23:59:23.407746 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399465 5274 flags.go:64] FLAG: --pods-per-core="0" Oct 30 23:59:23.407746 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399468 5274 flags.go:64] FLAG: --port="10250" Oct 30 23:59:23.407746 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399471 5274 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 23:59:23.407746 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399474 5274 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-09892ea9ffec88144" Oct 30 23:59:23.407746 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399478 5274 flags.go:64] FLAG: --qos-reserved="" Oct 30 23:59:23.407746 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399481 5274 flags.go:64] FLAG: --read-only-port="10255" Oct 30 23:59:23.407746 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399484 5274 flags.go:64] FLAG: --register-node="true" Oct 30 23:59:23.407746 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399488 5274 flags.go:64] FLAG: --register-with-taints="" Oct 30 23:59:23.407746 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399492 5274 flags.go:64] FLAG: --registry-burst="10" Oct 30 23:59:23.407746 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399495 5274 flags.go:64] FLAG: --registry-qps="5" Oct 30 23:59:23.407746 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399498 5274 flags.go:64] FLAG: --reserved-cpus="" Oct 30 23:59:23.407746 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399501 5274 flags.go:64] FLAG: --reserved-memory="" Oct 30 23:59:23.407746 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399505 5274 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 23:59:23.407746 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399508 5274 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 23:59:23.407746 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399511 5274 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 23:59:23.407746 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399514 5274 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 23:59:23.407746 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399519 5274 flags.go:64] FLAG: --runonce="false" Oct 30 23:59:23.408466 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399522 5274 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 23:59:23.408466 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399525 5274 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 23:59:23.408466 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399529 5274 flags.go:64] FLAG: --seccomp-default="false" Oct 30 23:59:23.408466 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399531 5274 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 23:59:23.408466 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399534 5274 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 23:59:23.408466 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399538 5274 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 23:59:23.408466 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399541 5274 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 23:59:23.408466 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399544 5274 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 23:59:23.408466 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399547 5274 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 23:59:23.408466 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399550 5274 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 23:59:23.408466 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399553 5274 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 23:59:23.408466 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399556 5274 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 23:59:23.408466 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399559 5274 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 23:59:23.408466 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399562 5274 flags.go:64] FLAG: --system-cgroups="" Oct 30 23:59:23.408466 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399566 5274 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 23:59:23.408466 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399571 5274 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 23:59:23.408466 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399574 5274 flags.go:64] FLAG: --tls-cert-file="" Oct 30 23:59:23.408466 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399577 5274 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 23:59:23.408466 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399582 5274 flags.go:64] FLAG: --tls-min-version="" Oct 30 23:59:23.408466 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399584 5274 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 23:59:23.408466 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399587 5274 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 23:59:23.408466 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399591 5274 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 23:59:23.408466 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399595 5274 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 23:59:23.408466 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399598 5274 flags.go:64] FLAG: --v="2" Oct 30 23:59:23.408466 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399602 5274 flags.go:64] FLAG: --version="false" Oct 30 23:59:23.409192 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399607 5274 flags.go:64] FLAG: --vmodule="" Oct 30 23:59:23.409192 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399611 5274 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 23:59:23.409192 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.399614 5274 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 23:59:23.409192 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399756 5274 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:59:23.409192 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399760 5274 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:59:23.409192 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399763 5274 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:59:23.409192 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399767 5274 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:59:23.409192 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399769 5274 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:59:23.409192 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399772 5274 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:59:23.409192 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399775 5274 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:59:23.409192 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399778 5274 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:59:23.409192 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399781 5274 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:59:23.409192 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399783 5274 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:59:23.409192 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399786 5274 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:59:23.409192 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399789 5274 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:59:23.409192 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399791 5274 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:59:23.409192 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399794 5274 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:59:23.409192 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399797 5274 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:59:23.409192 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399799 5274 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:59:23.409192 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399802 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:59:23.409679 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399805 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:59:23.409679 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399807 5274 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:59:23.409679 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399811 5274 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:59:23.409679 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399813 5274 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:59:23.409679 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399816 5274 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:59:23.409679 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399819 5274 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:59:23.409679 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399821 5274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:59:23.409679 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399824 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:59:23.409679 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399826 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:59:23.409679 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399831 5274 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:59:23.409679 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399833 5274 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:59:23.409679 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399836 5274 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:59:23.409679 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399838 5274 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:59:23.409679 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399841 5274 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:59:23.409679 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399844 5274 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:59:23.409679 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399846 5274 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:59:23.409679 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399849 5274 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:59:23.409679 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399853 5274 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:59:23.409679 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399857 5274 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:59:23.409679 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399861 5274 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:59:23.410239 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399864 5274 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:59:23.410239 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399867 5274 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:59:23.410239 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399870 5274 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:59:23.410239 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399872 5274 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:59:23.410239 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399875 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:59:23.410239 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399877 5274 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:59:23.410239 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399880 5274 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:59:23.410239 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399882 5274 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:59:23.410239 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399886 5274 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:59:23.410239 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399888 5274 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:59:23.410239 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399891 5274 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:59:23.410239 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399893 5274 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:59:23.410239 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399896 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:59:23.410239 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399898 5274 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:59:23.410239 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399901 5274 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:59:23.410239 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399904 5274 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:59:23.410239 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399907 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:59:23.410239 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399909 5274 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:59:23.410239 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399912 5274 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:59:23.410824 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399915 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:59:23.410824 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399917 5274 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:59:23.410824 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399921 5274 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:59:23.410824 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399924 5274 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:59:23.410824 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399927 5274 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:59:23.410824 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399929 5274 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:59:23.410824 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399932 5274 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:59:23.410824 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399934 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:59:23.410824 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399937 5274 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:59:23.410824 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399940 5274 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:59:23.410824 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399942 5274 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:59:23.410824 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399946 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:59:23.410824 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399948 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:59:23.410824 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399966 5274 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:59:23.410824 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399970 5274 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:59:23.410824 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399973 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:59:23.410824 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399975 5274 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:59:23.410824 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399978 5274 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:59:23.410824 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399980 5274 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:59:23.411385 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399983 5274 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:59:23.411385 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399986 5274 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:59:23.411385 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399989 5274 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:59:23.411385 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399992 5274 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:59:23.411385 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399994 5274 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:59:23.411385 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.399997 5274 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:59:23.411385 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400000 5274 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:59:23.411385 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400002 5274 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:59:23.411385 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400005 5274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:59:23.411385 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400008 5274 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:59:23.411385 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400011 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:59:23.411385 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400014 5274 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:59:23.411385 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400017 5274 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:59:23.411385 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400019 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:59:23.411385 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400022 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:59:23.411385 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400027 5274 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:59:23.411385 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400031 5274 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:59:23.411385 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400034 5274 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:59:23.411385 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400037 5274 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:59:23.412285 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400040 5274 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:59:23.412285 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400042 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:59:23.412285 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400044 5274 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:59:23.412285 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400047 5274 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:59:23.412285 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400050 5274 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:59:23.412285 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400052 5274 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:59:23.412285 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400055 5274 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:59:23.412285 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400057 5274 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:59:23.412285 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400060 5274 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:59:23.412285 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400063 5274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:59:23.412285 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400066 5274 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:59:23.412285 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400068 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:59:23.412285 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.400094 5274 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:59:23.412285 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400135 5274 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:59:23.412285 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400141 5274 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:59:23.412285 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400144 5274 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:59:23.413035 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400146 5274 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:59:23.413035 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400149 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:59:23.413035 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400152 5274 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:59:23.413035 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400155 5274 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:59:23.413035 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400159 5274 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:59:23.413035 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400161 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:59:23.413035 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400164 5274 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:59:23.413035 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400167 5274 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:59:23.413035 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400170 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:59:23.413035 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400173 5274 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:59:23.413035 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400176 5274 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:59:23.413035 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400179 5274 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:59:23.413035 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400183 5274 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:59:23.413035 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400185 5274 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:59:23.413035 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400188 5274 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:59:23.413035 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400190 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:59:23.413035 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400193 5274 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:59:23.413035 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400196 5274 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:59:23.413035 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400199 5274 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:59:23.413035 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400201 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:59:23.413608 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400204 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:59:23.413608 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400207 5274 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:59:23.413608 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400209 5274 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:59:23.413608 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400212 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:59:23.413608 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400215 5274 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:59:23.413608 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400217 5274 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:59:23.413608 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400220 5274 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:59:23.413608 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400223 5274 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:59:23.413608 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400226 5274 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:59:23.413608 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400228 5274 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:59:23.413608 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400232 5274 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:59:23.413608 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400235 5274 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:59:23.413608 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400237 5274 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:59:23.413608 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400240 5274 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:59:23.413608 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400243 5274 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:59:23.413608 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400246 5274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:59:23.413608 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400249 5274 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:59:23.413608 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400252 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:59:23.413608 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400255 5274 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:59:23.413608 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400258 5274 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:59:23.414252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400261 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:59:23.414252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400267 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:59:23.414252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400271 5274 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:59:23.414252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400275 5274 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:59:23.414252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400279 5274 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:59:23.414252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400282 5274 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:59:23.414252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400285 5274 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:59:23.414252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400287 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:59:23.414252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400290 5274 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:59:23.414252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400293 5274 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:59:23.414252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400295 5274 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:59:23.414252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400298 5274 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:59:23.414252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400300 5274 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:59:23.414252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400303 5274 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:59:23.414252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400306 5274 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:59:23.414252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400308 5274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:59:23.414252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400311 5274 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:59:23.414252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400314 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:59:23.414252 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400317 5274 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:59:23.414729 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400320 5274 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:59:23.414729 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400323 5274 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:59:23.414729 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400325 5274 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:59:23.414729 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400329 5274 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:59:23.414729 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400332 5274 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:59:23.414729 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400335 5274 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:59:23.414729 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400337 5274 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:59:23.414729 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400340 5274 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:59:23.414729 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400343 5274 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:59:23.414729 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400345 5274 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:59:23.414729 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400348 5274 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:59:23.414729 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400350 5274 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:59:23.414729 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400353 5274 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:59:23.414729 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400356 5274 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:59:23.414729 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400358 5274 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:59:23.414729 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400361 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:59:23.414729 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400365 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:59:23.414729 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400369 5274 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:59:23.414729 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400372 5274 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:59:23.414729 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400374 5274 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:59:23.415229 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400377 5274 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:59:23.415229 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400380 5274 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:59:23.415229 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400383 5274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:59:23.415229 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400385 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:59:23.415229 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400388 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:59:23.415229 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400390 5274 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:59:23.415229 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400393 5274 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:59:23.415229 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400395 5274 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:59:23.415229 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400398 5274 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:59:23.415229 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400401 5274 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:59:23.415229 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400403 5274 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:59:23.415229 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400406 5274 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:59:23.415229 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400409 5274 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:59:23.415229 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400413 5274 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:59:23.415229 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400417 5274 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:59:23.415229 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400422 5274 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:59:23.415229 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400425 5274 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:59:23.415229 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400428 5274 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:59:23.415229 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400431 5274 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:59:23.415229 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400434 5274 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:59:23.415704 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400437 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:59:23.415704 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400440 5274 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:59:23.415704 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400442 5274 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:59:23.415704 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.400445 5274 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:59:23.415704 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.404907 5274 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 23:59:23.415704 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.404945 5274 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 23:59:23.415704 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.404990 5274 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 23:59:23.415704 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.404998 5274 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 23:59:23.415704 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405187 5274 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:59:23.415704 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405197 5274 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:59:23.415704 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405204 5274 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:59:23.415704 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405210 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:59:23.415704 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405215 5274 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:59:23.415704 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405220 5274 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:59:23.415704 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405224 5274 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:59:23.415704 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405229 5274 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:59:23.415704 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405233 5274 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:59:23.415704 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405244 5274 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:59:23.415704 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405249 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:59:23.416175 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405253 5274 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:59:23.416175 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405257 5274 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:59:23.416175 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405262 5274 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:59:23.416175 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405266 5274 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:59:23.416175 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405271 5274 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:59:23.416175 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405275 5274 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:59:23.416175 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405279 5274 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:59:23.416175 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405284 5274 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:59:23.416175 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405288 5274 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:59:23.416175 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405292 5274 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:59:23.416175 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405297 5274 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:59:23.416175 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405306 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:59:23.416175 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405311 5274 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:59:23.416175 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405315 5274 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:59:23.416175 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405320 5274 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:59:23.416175 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405365 5274 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:59:23.416175 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405407 5274 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:59:23.416175 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405413 5274 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:59:23.416175 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405558 5274 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:59:23.416175 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405561 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:59:23.416660 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405566 5274 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:59:23.416660 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405569 5274 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:59:23.416660 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405572 5274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:59:23.416660 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405575 5274 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:59:23.416660 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405578 5274 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:59:23.416660 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405581 5274 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:59:23.416660 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405584 5274 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:59:23.416660 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405586 5274 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:59:23.416660 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405589 5274 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:59:23.416660 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405592 5274 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:59:23.416660 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405595 5274 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:59:23.416660 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405597 5274 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:59:23.416660 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405600 5274 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:59:23.416660 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405603 5274 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:59:23.416660 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405605 5274 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:59:23.416660 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405608 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:59:23.416660 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405613 5274 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:59:23.416660 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405616 5274 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:59:23.416660 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405619 5274 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:59:23.417134 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405622 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:59:23.417134 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405624 5274 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:59:23.417134 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405628 5274 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:59:23.417134 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405631 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:59:23.417134 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405633 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:59:23.417134 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405636 5274 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:59:23.417134 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405640 5274 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:59:23.417134 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405643 5274 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:59:23.417134 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405647 5274 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:59:23.417134 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405650 5274 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:59:23.417134 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405654 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:59:23.417134 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405657 5274 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:59:23.417134 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405660 5274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:59:23.417134 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405663 5274 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:59:23.417134 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405665 5274 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:59:23.417134 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405669 5274 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:59:23.417134 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405671 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:59:23.417134 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405674 5274 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:59:23.417134 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405677 5274 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:59:23.417134 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405680 5274 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:59:23.417607 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405682 5274 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:59:23.417607 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405685 5274 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:59:23.417607 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405688 5274 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:59:23.417607 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405690 5274 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:59:23.417607 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405693 5274 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:59:23.417607 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405697 5274 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:59:23.417607 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405700 5274 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:59:23.417607 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405703 5274 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:59:23.417607 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405705 5274 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:59:23.417607 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405708 5274 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:59:23.417607 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405710 5274 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:59:23.417607 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405713 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:59:23.417607 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405716 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:59:23.417607 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405721 5274 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:59:23.417607 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405723 5274 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:59:23.417607 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405726 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:59:23.417607 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405729 5274 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:59:23.417607 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405732 5274 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:59:23.417607 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405735 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:59:23.417607 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405737 5274 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:59:23.418205 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405740 5274 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:59:23.418205 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405743 5274 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:59:23.418205 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405746 5274 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:59:23.418205 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405748 5274 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:59:23.418205 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405751 5274 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:59:23.418205 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405753 5274 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:59:23.418205 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405756 5274 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:59:23.418205 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405759 5274 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:59:23.418205 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405762 5274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:59:23.418205 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405764 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:59:23.418205 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405767 5274 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:59:23.418205 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405772 5274 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:59:23.418205 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405777 5274 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:59:23.418205 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405780 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:59:23.418205 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405783 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:59:23.418205 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.405786 5274 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:59:23.418586 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.405826 5274 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:59:23.418586 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406032 5274 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:59:23.418586 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406039 5274 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:59:23.418586 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406042 5274 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:59:23.418586 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406045 5274 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:59:23.418586 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406048 5274 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:59:23.418586 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406053 5274 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:59:23.418586 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406056 5274 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:59:23.418586 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406059 5274 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:59:23.418586 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406062 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:59:23.418586 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406065 5274 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:59:23.418586 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406068 5274 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:59:23.418586 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406070 5274 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:59:23.418586 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406073 5274 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:59:23.418586 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406076 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:59:23.418586 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406079 5274 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:59:23.418975 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406082 5274 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:59:23.418975 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406085 5274 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:59:23.418975 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406088 5274 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:59:23.418975 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406091 5274 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:59:23.418975 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406093 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:59:23.418975 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406096 5274 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:59:23.418975 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406099 5274 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:59:23.418975 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406101 5274 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:59:23.418975 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406105 5274 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:59:23.418975 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406108 5274 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:59:23.418975 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406110 5274 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:59:23.418975 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406113 5274 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:59:23.418975 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406116 5274 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:59:23.418975 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406118 5274 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:59:23.418975 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406121 5274 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:59:23.418975 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406124 5274 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:59:23.418975 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406126 5274 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:59:23.418975 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406129 5274 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:59:23.418975 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406132 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:59:23.419431 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406135 5274 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:59:23.419431 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406137 5274 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:59:23.419431 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406140 5274 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:59:23.419431 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406144 5274 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:59:23.419431 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406149 5274 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:59:23.419431 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406152 5274 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:59:23.419431 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406154 5274 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:59:23.419431 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406157 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:59:23.419431 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406159 5274 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:59:23.419431 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406162 5274 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:59:23.419431 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406165 5274 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:59:23.419431 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406168 5274 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:59:23.419431 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406170 5274 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:59:23.419431 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406173 5274 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:59:23.419431 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406175 5274 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:59:23.419431 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406178 5274 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:59:23.419431 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406181 5274 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:59:23.419431 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406183 5274 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:59:23.419431 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406186 5274 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:59:23.419903 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406188 5274 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:59:23.419903 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406191 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:59:23.419903 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406193 5274 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:59:23.419903 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406196 5274 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:59:23.419903 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406199 5274 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:59:23.419903 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406202 5274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:59:23.419903 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406204 5274 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:59:23.419903 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406207 5274 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:59:23.419903 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406209 5274 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:59:23.419903 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406212 5274 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:59:23.419903 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406215 5274 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:59:23.419903 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406217 5274 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:59:23.419903 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406220 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:59:23.419903 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406222 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:59:23.419903 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406225 5274 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:59:23.419903 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406228 5274 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:59:23.419903 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406230 5274 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:59:23.419903 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406233 5274 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:59:23.419903 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406237 5274 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:59:23.419903 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406239 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:59:23.420388 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406242 5274 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:59:23.420388 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406245 5274 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:59:23.420388 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406247 5274 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:59:23.420388 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406250 5274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:59:23.420388 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406252 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:59:23.420388 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406255 5274 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:59:23.420388 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406258 5274 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:59:23.420388 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406260 5274 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:59:23.420388 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406263 5274 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:59:23.420388 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406266 5274 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:59:23.420388 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406268 5274 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:59:23.420388 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406271 5274 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:59:23.420388 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406275 5274 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:59:23.420388 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406278 5274 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:59:23.420388 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406281 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:59:23.420388 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406284 5274 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:59:23.420388 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406287 5274 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:59:23.420388 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406290 5274 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:59:23.420388 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406292 5274 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:59:23.420846 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406295 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:59:23.420846 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406298 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:59:23.420846 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406300 5274 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:59:23.420846 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406303 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:59:23.420846 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406305 5274 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:59:23.420846 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406308 5274 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:59:23.420846 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406311 5274 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:59:23.420846 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406313 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:59:23.420846 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406316 5274 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:59:23.420846 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406318 5274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:59:23.420846 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406321 5274 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:59:23.420846 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406323 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:59:23.420846 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406326 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:59:23.420846 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406329 5274 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:59:23.420846 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.406355 5274 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:59:23.421459 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406388 5274 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:59:23.421459 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406392 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:59:23.421459 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406395 5274 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:59:23.421459 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406399 5274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:59:23.421459 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406451 5274 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:59:23.421459 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406454 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:59:23.421459 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406458 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:59:23.421459 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406461 5274 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:59:23.421459 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406464 5274 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:59:23.421459 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406466 5274 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:59:23.421459 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406469 5274 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:59:23.421459 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406472 5274 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:59:23.421459 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406475 5274 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:59:23.421459 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406477 5274 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:59:23.421459 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406480 5274 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:59:23.421459 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406482 5274 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:59:23.421459 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406485 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:59:23.421459 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406488 5274 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:59:23.421459 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406491 5274 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:59:23.422366 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406494 5274 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:59:23.422366 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406496 5274 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:59:23.422366 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406499 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:59:23.422366 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406501 5274 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:59:23.422366 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406505 5274 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:59:23.422366 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406507 5274 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:59:23.422366 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406510 5274 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:59:23.422366 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406512 5274 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:59:23.422366 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406515 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:59:23.422366 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406517 5274 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:59:23.422366 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406520 5274 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:59:23.422366 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406524 5274 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:59:23.422366 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406527 5274 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:59:23.422366 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406533 5274 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:59:23.422366 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406536 5274 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:59:23.422366 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406539 5274 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:59:23.422366 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406541 5274 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:59:23.422366 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406544 5274 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:59:23.422366 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406546 5274 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:59:23.422366 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406549 5274 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:59:23.423032 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406552 5274 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:59:23.423032 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406554 5274 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:59:23.423032 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406557 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:59:23.423032 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406560 5274 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:59:23.423032 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406563 5274 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:59:23.423032 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406567 5274 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:59:23.423032 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406571 5274 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:59:23.423032 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406575 5274 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:59:23.423032 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406578 5274 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:59:23.423032 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406581 5274 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:59:23.423032 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406583 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:59:23.423032 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406586 5274 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:59:23.423032 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406589 5274 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:59:23.423032 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406591 5274 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:59:23.423032 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406594 5274 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:59:23.423032 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406597 5274 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:59:23.423032 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406599 5274 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:59:23.423032 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406602 5274 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:59:23.423032 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406605 5274 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:59:23.423502 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406607 5274 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:59:23.423502 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406610 5274 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:59:23.423502 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406613 5274 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:59:23.423502 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406615 5274 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:59:23.423502 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406619 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:59:23.423502 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406623 5274 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:59:23.423502 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406627 5274 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:59:23.423502 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406630 5274 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:59:23.423502 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406632 5274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:59:23.423502 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406635 5274 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:59:23.423502 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406638 5274 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:59:23.423502 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406640 5274 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:59:23.423502 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406643 5274 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:59:23.423502 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406645 5274 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:59:23.423502 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406648 5274 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:59:23.423502 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406650 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:59:23.423502 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406653 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:59:23.423502 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406656 5274 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:59:23.423502 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406658 5274 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:59:23.423502 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406660 5274 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:59:23.424003 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406663 5274 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:59:23.424003 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406666 5274 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:59:23.424003 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406669 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:59:23.424003 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406671 5274 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:59:23.424003 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406674 5274 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:59:23.424003 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406676 5274 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:59:23.424003 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406679 5274 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:59:23.424003 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406681 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:59:23.424003 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406684 5274 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:59:23.424003 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406686 5274 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:59:23.424003 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406689 5274 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:59:23.424003 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406692 5274 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:59:23.424003 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406694 5274 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:59:23.424003 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406697 5274 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:59:23.424003 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406699 5274 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:59:23.424003 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406703 5274 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:59:23.424003 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406708 5274 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:59:23.424003 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406711 5274 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:59:23.424003 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406715 5274 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:59:23.424003 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406718 5274 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:59:23.424535 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406720 5274 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:59:23.424535 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406723 5274 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:59:23.424535 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406727 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:59:23.424535 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406730 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:59:23.424535 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406733 5274 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:59:23.424535 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406735 5274 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:59:23.424535 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406738 5274 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:59:23.424535 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:23.406740 5274 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:59:23.424535 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.406807 5274 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 23:59:23.424535 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.407879 5274 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 23:59:23.424535 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.407945 5274 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 23:59:23.424535 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.408146 5274 server.go:1014] "Starting client certificate rotation" Oct 30 23:59:23.424535 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.408291 5274 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 23:59:23.424535 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.408344 5274 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 23:53:13 +0000 UTC" deadline="2028-04-02 11:19:14.475133829 +0000 UTC" Oct 30 23:59:23.424535 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.408383 5274 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12467h19m51.066753297s" Oct 30 23:59:23.424535 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.408624 5274 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 23:59:23.424968 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.409735 5274 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 23:59:23.424968 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.410992 5274 log.go:25] "Validated CRI v1 runtime API" Oct 30 23:59:23.424968 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.413291 5274 log.go:25] "Validated CRI v1 image API" Oct 30 23:59:23.424968 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.414041 5274 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 23:59:23.424968 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.416000 5274 fs.go:135] Filesystem UUIDs: map[4c9d44f2-3e4d-4627-a35a-38d3ce7417fd:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d267168e-f8ab-4cee-827b-d48137613b26:/dev/nvme0n1p3] Oct 30 23:59:23.425211 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.416023 5274 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/04a789b9af69c8958d287e296d45a9a03c561c67ef9faed9c7ab80d450c456c9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/04a789b9af69c8958d287e296d45a9a03c561c67ef9faed9c7ab80d450c456c9/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a478aca183c6cfef37fda65497616cbc096d1e47640b0f003ca2605afba647e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a478aca183c6cfef37fda65497616cbc096d1e47640b0f003ca2605afba647e/userdata/shm major:0 minor:261 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2c4dfef2498ce2f91228a57d0ac1e3a8e6124e8c5c0694400179b595b0aade48/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2c4dfef2498ce2f91228a57d0ac1e3a8e6124e8c5c0694400179b595b0aade48/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/34539fa69c119c636d1acc43e480f7606c57d5d847b5c124491b8ea368c84256/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/34539fa69c119c636d1acc43e480f7606c57d5d847b5c124491b8ea368c84256/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/397510d0f15462f0e47d89b988695a3d37f74841306435abd01856455179c4c5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/397510d0f15462f0e47d89b988695a3d37f74841306435abd01856455179c4c5/userdata/shm major:0 minor:359 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/409e18ce6451a89d480abb89b9d4c3cf0655defaeda3960aec5bffe99a10a7fc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/409e18ce6451a89d480abb89b9d4c3cf0655defaeda3960aec5bffe99a10a7fc/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/41dc65a16bae60af4f6fc3d44fe15f521d3a3580053e347ceb1ee9f5339e1749/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/41dc65a16bae60af4f6fc3d44fe15f521d3a3580053e347ceb1ee9f5339e1749/userdata/shm major:0 minor:355 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/50782c2accb92735384b1ec75130c51e641b2414a95d55743e69bc02be3519ca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/50782c2accb92735384b1ec75130c51e641b2414a95d55743e69bc02be3519ca/userdata/shm major:0 minor:317 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/707fc65a7121853641a6a4818f86b0ff658108656a1fe93eae4ab2f9b5e8fea1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/707fc65a7121853641a6a4818f86b0ff658108656a1fe93eae4ab2f9b5e8fea1/userdata/shm major:0 minor:357 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7ed7577cf3a690a028a8810df75503f3a7aaa9b5ee22832c2944fd235487c84e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7ed7577cf3a690a028a8810df75503f3a7aaa9b5ee22832c2944fd235487c84e/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8bf9deef04f1f887138a935ae7aa51b824efe2c52092d0c3d2111a2af206d09d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8bf9deef04f1f887138a935ae7aa51b824efe2c52092d0c3d2111a2af206d09d/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/92470d7928996059c02f744bd144c8e341b169b6fc78e28dcb0746dd804aee25/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/92470d7928996059c02f744bd144c8e341b169b6fc78e28dcb0746dd804aee25/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a01b5325ff7f82a3e316636bffb553afc1f5fa152a18c4ec81a031759ec90dad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a01b5325ff7f82a3e316636bffb553afc1f5fa152a18c4ec81a031759ec90dad/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a5b7fb8850839f327a21ed0ca1950d02333706f22908cae2ae7631d85e53f37b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a5b7fb8850839f327a21ed0ca1950d02333706f22908cae2ae7631d85e53f37b/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b3dd912a576bc95f147b1188d016ea64820726576094e8de17e26a806ea21706/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b3dd912a576bc95f147b1188d016ea64820726576094e8de17e26a806ea21706/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b665f50eb4a136f42a87c03c99aec73c08df17ec99489f998035c4252173865f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b665f50eb4a136f42a87c03c99aec73c08df17ec99489f998035c4252173865f/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bbef4aa45643805424f941e549c15f58ddc8ab57981152c689444be36ab768e7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bbef4aa45643805424f941e549c15f58ddc8ab57981152c689444be36ab768e7/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cca062ee4fe9f407fdab48547ae139d0d1957b3acbbce1c68840c495f196fd4a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cca062ee4fe9f407fdab48547ae139d0d1957b3acbbce1c68840c495f196fd4a/userdata/shm major:0 minor:326 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ced632c8d7bead7d8d4c32eb34db763bdb44e1c743c8ea92132b98980ca46cad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ced632c8d7bead7d8d4c32eb34db763bdb44e1c743c8ea92132b98980ca46cad/userdata/shm major:0 minor:263 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dbe38159a2b22ae460f368a9a9408a07e42e1078e30811ab06e7c16edd791eb9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dbe38159a2b22ae460f368a9a9408a07e42e1078e30811ab06e7c16edd791eb9/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fe4d0060951c6651063102dcb213c97f5b3d2d38e0cc975985f5dedd64be8cf4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fe4d0060951c6651063102dcb213c97f5b3d2d38e0cc975985f5dedd64be8cf4/userdata/shm major:0 minor:259 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e/volumes/kubernetes.io~projected/kube-api-access-5fr52:{mountpoint:/var/lib/kubelet/pods/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e/volumes/kubernetes.io~projected/kube-api-access-5fr52 major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0adc098d-57a4-490e-89b7-b1b271a1c087/volumes/kubernetes.io~projected/kube-api-access-djj7v:{mountpoint:/var/lib/kubelet/pods/0adc098d-57a4-490e-89b7-b1b271a1c087/volumes/kubernetes.io~projected/kube-api-access-djj7v major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0adc098d-57a4-490e-89b7-b1b271a1c087/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/0adc098d-57a4-490e-89b7-b1b271a1c087/volumes/kubernetes.io~secret/metrics-certs major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18698607-3254-4ec9-b3bb-bbe59c6c744f/volumes/kubernetes.io~projected/kube-api-access-cqkgh:{mountpoint:/var/lib/kubelet/pods/18698607-3254-4ec9-b3bb-bbe59c6c744f/volumes/kubernetes.io~projected/kube-api-access-cqkgh major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18698607-3254-4ec9-b3bb-bbe59c6c744f/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/18698607-3254-4ec9-b3bb-bbe59c6c744f/volumes/kubernetes.io~secret/cert major:0 minor:196 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18acd0a8-988e-4168-84e7-0b25dc7b32d4/volumes/kubernetes.io~projected/kube-api-access-pmcgf:{mountpoint:/var/lib/kubelet/pods/18acd0a8-988e-4168-84e7-0b25dc7b32d4/volumes/kubernetes.io~projected/kube-api-access-pmcgf major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ad2242c-28d7-4495-9bac-f6c8cba7f796/volumes/kubernetes.io~projected/kube-api-access-gprnc:{mountpoint:/var/lib/kubelet/pods/1ad2242c-28d7-4495-9bac-f6c8cba7f796/volumes/kubernetes.io~projected/kube-api-access-gprnc major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ad2242c-28d7-4495-9bac-f6c8cba7f796/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/1ad2242c-28d7-4495-9bac-f6c8cba7f796/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ad2242c-28d7-4495-9bac-f6c8cba7f796/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/1ad2242c-28d7-4495-9bac-f6c8cba7f796/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f653a44-907a-417e-a1da-9b1bb0ee7283/volumes/kubernetes.io~projected/kube-api-access-r8tds:{mountpoint:/var/lib/kubelet/pods/1f653a44-907a-417e-a1da-9b1bb0ee7283/volumes/kubernetes.io~projected/kube-api-access-r8tds major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25e99515-222f-4346-9fe5-5b7b3b213ee8/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/25e99515-222f-4346-9fe5-5b7b3b213ee8/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25e99515-222f-4346-9fe5-5b7b3b213ee8/volumes/kubernetes.io~projected/kube-api-access-dpxpg:{mountpoint:/var/lib/kubelet/pods/25e99515-222f-4346-9fe5-5b7b3b213ee8/volumes/kubernetes.io~projected/kube-api-access-dpxpg major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25e99515-222f-4346-9fe5-5b7b3b213ee8/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/25e99515-222f-4346-9fe5-5b7b3b213ee8/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3cb65abb-ba0e-4075-9a0a-455e1b64b820/volumes/kubernetes.io~projected/kube-api-access-wsblw:{mountpoint:/var/lib/kubelet/pods/3cb65abb-ba0e-4075-9a0a-455e1b64b820/volumes/kubernetes.io~projected/kube-api-access-wsblw major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3cb65abb-ba0e-4075-9a0a-455e1b64b820/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/3cb65abb-ba0e-4075-9a0a-455e1b64b820/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/409b7296-1d35-44e9-be1a-3459f0dbaad9/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/409b7296-1d35-44e9-be1a-3459f0dbaad9/volumes/kubernetes.io~secret/agent-certs major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50c15c32-e5cd-426c-9e7d-c123f8480653/volumes/kubernetes.io~projected/kube-api-access-gd4j2:{mountpoint:/var/lib/kubelet/pods/50c15c32-e5cd-426c-9e7d-c123f8480653/volumes/kubernetes.io~projected/kube-api-access-gd4j2 major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5405bfc2-cfc0-4ea6-80bc-911e117f1843/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/5405bfc2-cfc0-4ea6-80bc-911e117f1843/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5405bfc2-cfc0-4ea6-80bc-911e117f1843/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/5405bfc2-cfc0-4ea6-80bc-911e117f1843/volumes/kubernetes.io~empty-dir/tmp major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5405bfc2-cfc0-4ea6-80bc-911e117f1843/volumes/kubernetes.io~projected/kube-api-access-mcvd6:{mountpoint:/var/lib/kubelet/pods/5405bfc2-cfc0-4ea6-80bc-911e117f1843/volumes/kubernetes.io~projected/kube-api-access-mcvd6 major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ab2f199-64ea-4071-b8cd-6b9861d31179/volumes/kubernetes.io~projected/kube-api-access-qfgxg:{mountpoint:/var/lib/kubelet/pods/5ab2f199-64ea-4071-b8cd-6b9861d31179/volumes/kubernetes.io~projected/kube-api-access-qfgxg major:0 minor:256 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ab2f199-64ea-4071-b8cd-6b9861d31179/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/5ab2f199-64ea-4071-b8cd-6b9861d31179/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ab2f199-64ea-4071-b8cd-6b9861d31179/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/5ab2f199-64ea-4071-b8cd-6b9861d31179/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63e868ff-3f36-4542-9e1d-9688ce8c37f1/volumes/kubernetes.io~projected/kube-api-access-fl2s5:{mountpoint:/var/lib/kubelet/pods/63e868ff-3f36-4542-9e1d-9688ce8c37f1/volumes/kubernetes.io~projected/kube-api-access-fl2s5 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b099593-ded8-475c-b5ef-a91bbd9aeb1a/volumes/kubernetes.io~projected/kube-api-access-gnlg7:{mountpoint:/var/lib/kubelet/pods/8b099593-ded8-475c-b5ef-a91bbd9aeb1a/volumes/kubernetes.io~projected/kube-api-access-gnlg7 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b099593-ded8-475c-b5ef-a91bbd9aeb1a/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/8b099593-ded8-475c-b5ef-a91bbd9aeb1a/volumes/kubernetes.io~secret/metrics-tls major:0 minor:200 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99b4d92d-2003-4708-99da-a5f12cbb10e2/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/99b4d92d-2003-4708-99da-a5f12cbb10e2/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0be3213-788c-45b4-8fbd-37e744a14f17/volumes/kubernetes.io~projected/kube-api-access-hlqqp:{mountpoint:/var/lib/kubelet/pods/a0be3213-788c-45b4-8fbd-37e744a14f17/volumes/kubernetes.io~projected/kube-api-access-hlqqp major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8d0e1f0-620a-46c0-8234-628fb8acb572/volumes/kubernetes.io~projected/kube-api-access-znq6p:{mountpoint:/var/lib/kubelet/pods/c8d0e1f0-620a-46c0-8234-628fb8acb572/volumes/kubernetes.io~projected/kube-api-access-znq6p major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8d0e1f0-620a-46c0-8234-628fb8acb572/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/c8d0e1f0-620a-46c0-8234-628fb8acb572/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8d0e1f0-620a-46c0-8234-628fb8acb572/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/c8d0e1f0-620a-46c0-8234-628fb8acb572/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8d0e1f0-620a-46c0-8234-628fb8acb572/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/c8d0e1f0-620a-46c0-8234-628fb8acb572/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8d0e1f0-620a-46c0-8234-628fb8acb572/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/c8d0e1f0-620a-46c0-8234-628fb8acb572/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7815306-7a99-4fc5-8226-b34916ceaaa2/volumes/kubernetes.io~projected/kube-api-access-2kjxh:{mountpoint:/var/lib/kubelet/pods/d7815306-7a99-4fc5-8226-b34916ceaaa2/volumes/kubernetes.io~projected/kube-api-access-2kjxh major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8089437-f546-441e-b121-6e79878d2608/volumes/kubernetes.io~projected/kube-api-access-d8wrl:{mountpoint:/var/lib/kubelet/pods/d8089437-f546-441e-b121-6e79878d2608/volumes/kubernetes.io~projected/kube-api-access-d8wrl major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9/volumes/kubernetes.io~projected/kube-api-access-85v2m:{mountpoint:/var/lib/kubelet/pods/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9/volumes/kubernetes.io~projected/kube-api-access-85v2m major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:254 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/ef094b97b43b3cc42a4c377a4da521b2e0d7ef8f47f6c668cf670f5c4c000c38/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/0702a318c1d1ac10843a6f5c0c126bf5c1b02539f247ebc2deaa947f70cbb606/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/7f0509db38f2d1323880de781ba0c4d3df651b14c146d8763ac15e6bd9d4295e/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/b0bba52606f1f0e95671e9211ad8ef68937dc6b6f03277644d1e84da70aca08e/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/9f71b2d8b74a7cdddc92f566d86be6dd7c64a54ca216a3cad32b532daf52373a/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-116:{mountpoint:/var/lib/containers/storage/overlay/52059bff40965ca35379b8f20a821e50b4e9f7b709f5e8ba65a844b42036aced/merged major:0 minor:116 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/19192f41ea178405008d92ecd47c7ff2df3113ed55fad72b5784b5015c7b6887/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/4c054032941bd861ce5a01ef4fc9d9d1264cb89f13abdc3c22a8f8c1d74de031/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/2caf11a15ada831a47c4573ab28b6248cd96b7208b62301e313eeed239f351d5/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/b40d3e2a5a8e7c6ab5793c7415c3be386256aec1b1729bc97b60c06e0fe510cf/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/0ad52b6d4d5364b306ca8d067b44ea16eeecbc55ccb787a38bf4e153d46f8c28/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/80e335dbedbdaebe7ef50edba87405c5c6ae9d3e8a6f2562aeb01cdd73578241/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/f80b47f0d409d7c582d72f5242fbb4f8d150deeb4d8ed5dcbab6cb97658a6046/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/e67cd932a07e9aec03542aeef3f06f964cddb1bee52b44e22eb82dfcfbde8ff2/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/72f0bf5957f1bb53bef2d709b18f0a76af005ec86223790a6d5ae148c5fb3cb1/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/c29a262ead5d27d8a4442737f5834b46ec59684b38a4c494dc99cdab98ff7d95/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/451f7fd22c7c4e56f7347e3efd474d040e735e961dfd7f292d31380038d6adc9/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/e5c0a4724f0f0dd9d4bd8e65772df290311b3c8993cd030a62ef9bd68167e54b/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/1e290a4ce14bff4a91f45f8b044f1a6bad8a8b82d4dbe8f2b806ddb0233d0541/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/9e68f4b4d87288363a3e77df8091b5fb9ea7702bb7c648d689f501711b71f9e4/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/0750d7bb4e508ba05b74a670ac6f1f50d2e9d68b5220addfeb5e79138e0f04b5/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/5d6e7eefcee20e4d9b3d1614ab71ec4bc5869c2343b42fdb0bc84684f4e8af61/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/fbaaa6ce2ebbb2b8adc0717f66bf0487ef613af02bd4edec832277f84359fb93/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/d89241533320a69b851518b311f9057aad97185b6138fba91b6df7259948b3c9/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/a656b12f3d8c5bad79281e2b9296159f331912940a173f77b70c300c227efa92/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/b4868649586c43dea624bc60b6427242fd02f1d78969aeed33bfac4e549a740f/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/f330aa95f68893de0527e068584743a45e905759dda7be7ff90e29af5adc5e31/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/e111ab570976ba2113db9942cc1ef477a78b45dcfea07c2fa9dd9afc6169abc7/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/f952ea349c205fabb5105464302ebd049b81b21511b2bad6232efc3f33f75b4c/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/8c45ea53a36d0684c8692efca97110c0674bf4536f6d61a98d0998a62e93d577/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/d633fc95a895cb9be9817b5c0bfd1415a9215a49ecc133385d933da48a054d7b/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/317750f78cde70434074eaeb540c58ae2ba368aff0526730606f259a3832c74c/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/4e0b5a84cbe8d7885ec569a805e96ef6bb88a8370bb52db9c0874295a1dbe5e9/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/6320cda0624465a2168e91d65ac69c85bd4a2457ffa14034dd7e57125d764dee/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/fc764b340cc51640bb7eb255b9d6fb7f65f53512d32e8dafdec875ea2bbb01ea/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/ba870e1effadaa57dc4530d5494280bf17ecb27d0f62f1f155e8b30d4baa24e6/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/3278b840d285003040018ee3168867fed541178985bf683d049cfc6de7f71807/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/2a444f8ec2eb19e107d93173db81a2aa33f65df70208c3890bc61c185c221ef4/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/bd1ba274076d53184486fc2e461eb57025ec0b35d63a84ba38f91bc739edaba5/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/54cebde6c46d04cafc1e38d38aa8358e928c2b62904611ff7654f1e6cc0eac9d/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/5681886103fb9f4c70aac46284c331252f5c85da92c33b412c82a9301028cc5c/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/1578fea26b5aef1af5d2037ad0c5ee0a5a7be46973c01df2176b681b5a43645f/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/f4288f50c9694c0c1bdbc906d0f1276a734b28589baabb27ab78e354f6775c23/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/7ae00a3086043ff627661a17e90b5fb45ab3b87410fd5d697a2eb924e9aff984/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/909dff610b3b36b58dbc37dce0c3c4002159b3206d1b29ccf8de15c17c2e8e70/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/2fdef08c9ed83c61336f1c3fc15ae8e0180fc59405a04c53767e66b150323046/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/480025ef913641ed840783916676915ff987c2cbe05b3464440c6342482a0750/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/ddac78057e181a3658be94f4e933b995464281edc7e9587dd82657694c5fce96/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-365:{mountpoint:/var/lib/containers/storage/overlay/10f0f70aa20efa7bd8c87c8ee289cf5aa8b63eed7026948005e9899272d9f1c2/merged major:0 minor:365 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/5424071cc6fcfd83d9b9c1fd1ca9fb5979a74052ce25ed77b23b991b76da36a9/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-369:{mountpoint:/var/lib/containers/storage/overlay/358430ac50bc90ad5f823cdf272a57802a6a4c765ffd9f2c14338ebb3af3843f/merged major:0 minor:369 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/37ece44d89ab1d3954ac4d8ff4718eca2c507a01084dc8996cdc82f14a147483/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/64828d58ec2255f82623936936201365bad45cfd87913fb146b24dd8420c3645/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-45:{mountpoint:/var/lib/containers/storage/overlay/b418153fdc1e40369da62559dc5bdf4776940257c06d348c478cf7cd6a8cdc98/merged major:0 minor:45 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/0e1c890e0e5283151537de1ded4f08a42fd27d519a4df1b41e8602c84f4517f9/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/797b4e72fb3500fcbc0e07520c62b57807ef0e0a57d010ea60468644084d56e3/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/aeda056bfb3bd17816e0d4a7b89fe6d19f19c99b3ceb74e87cf520de34c2c30a/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/a752315cdd4ed28df7eb68a1fd58a87e7a3060c1686bb899e0669f9bc864eae8/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/e85dffb110db3c00f2155c28fe3f07c17c59ff3c78a0a310f538f86bdcd65771/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/9bd684a1e91c2b8eed6531841145845a62e02b0cd8a6659330450e03a2737ed6/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/a8b0bb6288b55529b9166f69190eb4345f5e278e70d2242af33c23dc0a3d71d5/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/67ce6824f0460903fa3b15969fb4fcd8b68a58e9b2cc9f2d48a8ac9f194e5916/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/ab199b36dd229fc4bb68ce0c39455b0a0cf44b4949902f96d0b51f925f954507/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/2278900c7488642c40b0f535ab2a43f3d1dbe3c1e783fa035ad9ab86c8268879/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 30 23:59:23.425565 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.423588 5274 manager.go:217] Machine: {Timestamp:2026-10-30 23:59:23.423121082 +0000 UTC m=+0.059939788 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099197 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24f91baf472d4c7b81fb5c645d05aa SystemUUID:ec24f91b-af47-2d4c-7b81-fb5c645d05aa BootID:c8f29756-d4ce-43e8-89a1-0ba0a1f6f014 Filesystems:[{Device:/run/containers/storage/overlay-containers/ced632c8d7bead7d8d4c32eb34db763bdb44e1c743c8ea92132b98980ca46cad/userdata/shm DeviceMajor:0 DeviceMinor:263 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1ad2242c-28d7-4495-9bac-f6c8cba7f796/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:253 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c8d0e1f0-620a-46c0-8234-628fb8acb572/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:321 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/707fc65a7121853641a6a4818f86b0ff658108656a1fe93eae4ab2f9b5e8fea1/userdata/shm DeviceMajor:0 DeviceMinor:357 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5405bfc2-cfc0-4ea6-80bc-911e117f1843/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0adc098d-57a4-490e-89b7-b1b271a1c087/volumes/kubernetes.io~projected/kube-api-access-djj7v DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5ab2f199-64ea-4071-b8cd-6b9861d31179/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:246 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cca062ee4fe9f407fdab48547ae139d0d1957b3acbbce1c68840c495f196fd4a/userdata/shm DeviceMajor:0 DeviceMinor:326 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1f653a44-907a-417e-a1da-9b1bb0ee7283/volumes/kubernetes.io~projected/kube-api-access-r8tds DeviceMajor:0 DeviceMinor:52 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a0be3213-788c-45b4-8fbd-37e744a14f17/volumes/kubernetes.io~projected/kube-api-access-hlqqp DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-45 DeviceMajor:0 DeviceMinor:45 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/25e99515-222f-4346-9fe5-5b7b3b213ee8/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5405bfc2-cfc0-4ea6-80bc-911e117f1843/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:51 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3cb65abb-ba0e-4075-9a0a-455e1b64b820/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:255 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9/volumes/kubernetes.io~projected/kube-api-access-85v2m DeviceMajor:0 DeviceMinor:258 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b665f50eb4a136f42a87c03c99aec73c08df17ec99489f998035c4252173865f/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bbef4aa45643805424f941e549c15f58ddc8ab57981152c689444be36ab768e7/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/63e868ff-3f36-4542-9e1d-9688ce8c37f1/volumes/kubernetes.io~projected/kube-api-access-fl2s5 DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8bf9deef04f1f887138a935ae7aa51b824efe2c52092d0c3d2111a2af206d09d/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b3dd912a576bc95f147b1188d016ea64820726576094e8de17e26a806ea21706/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7ed7577cf3a690a028a8810df75503f3a7aaa9b5ee22832c2944fd235487c84e/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/25e99515-222f-4346-9fe5-5b7b3b213ee8/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/5ab2f199-64ea-4071-b8cd-6b9861d31179/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:252 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/34539fa69c119c636d1acc43e480f7606c57d5d847b5c124491b8ea368c84256/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:314 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/41dc65a16bae60af4f6fc3d44fe15f521d3a3580053e347ceb1ee9f5339e1749/userdata/shm DeviceMajor:0 DeviceMinor:355 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/50c15c32-e5cd-426c-9e7d-c123f8480653/volumes/kubernetes.io~projected/kube-api-access-gd4j2 DeviceMajor:0 DeviceMinor:354 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-369 DeviceMajor:0 DeviceMinor:369 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1ad2242c-28d7-4495-9bac-f6c8cba7f796/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:251 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c8d0e1f0-620a-46c0-8234-628fb8acb572/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:322 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/397510d0f15462f0e47d89b988695a3d37f74841306435abd01856455179c4c5/userdata/shm DeviceMajor:0 DeviceMinor:359 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/25e99515-222f-4346-9fe5-5b7b3b213ee8/volumes/kubernetes.io~projected/kube-api-access-dpxpg DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b099593-ded8-475c-b5ef-a91bbd9aeb1a/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:200 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/18698607-3254-4ec9-b3bb-bbe59c6c744f/volumes/kubernetes.io~projected/kube-api-access-cqkgh DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:310 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c8d0e1f0-620a-46c0-8234-628fb8acb572/volumes/kubernetes.io~projected/kube-api-access-znq6p DeviceMajor:0 DeviceMinor:325 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/99b4d92d-2003-4708-99da-a5f12cbb10e2/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:353 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2c4dfef2498ce2f91228a57d0ac1e3a8e6124e8c5c0694400179b595b0aade48/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e/volumes/kubernetes.io~projected/kube-api-access-5fr52 DeviceMajor:0 DeviceMinor:316 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0adc098d-57a4-490e-89b7-b1b271a1c087/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:349 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5ab2f199-64ea-4071-b8cd-6b9861d31179/volumes/kubernetes.io~projected/kube-api-access-qfgxg DeviceMajor:0 DeviceMinor:256 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fe4d0060951c6651063102dcb213c97f5b3d2d38e0cc975985f5dedd64be8cf4/userdata/shm DeviceMajor:0 DeviceMinor:259 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5405bfc2-cfc0-4ea6-80bc-911e117f1843/volumes/kubernetes.io~projected/kube-api-access-mcvd6 DeviceMajor:0 DeviceMinor:53 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/18acd0a8-988e-4168-84e7-0b25dc7b32d4/volumes/kubernetes.io~projected/kube-api-access-pmcgf DeviceMajor:0 DeviceMinor:54 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/18698607-3254-4ec9-b3bb-bbe59c6c744f/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:196 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3cb65abb-ba0e-4075-9a0a-455e1b64b820/volumes/kubernetes.io~projected/kube-api-access-wsblw DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d7815306-7a99-4fc5-8226-b34916ceaaa2/volumes/kubernetes.io~projected/kube-api-access-2kjxh DeviceMajor:0 DeviceMinor:47 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/409b7296-1d35-44e9-be1a-3459f0dbaad9/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:50 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a01b5325ff7f82a3e316636bffb553afc1f5fa152a18c4ec81a031759ec90dad/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/409e18ce6451a89d480abb89b9d4c3cf0655defaeda3960aec5bffe99a10a7fc/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:315 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-116 DeviceMajor:0 DeviceMinor:116 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:254 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/8b099593-ded8-475c-b5ef-a91bbd9aeb1a/volumes/kubernetes.io~projected/kube-api-access-gnlg7 DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dbe38159a2b22ae460f368a9a9408a07e42e1078e30811ab06e7c16edd791eb9/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d8089437-f546-441e-b121-6e79878d2608/volumes/kubernetes.io~projected/kube-api-access-d8wrl DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a5b7fb8850839f327a21ed0ca1950d02333706f22908cae2ae7631d85e53f37b/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a478aca183c6cfef37fda65497616cbc096d1e47640b0f003ca2605afba647e/userdata/shm DeviceMajor:0 DeviceMinor:261 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/04a789b9af69c8958d287e296d45a9a03c561c67ef9faed9c7ab80d450c456c9/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1ad2242c-28d7-4495-9bac-f6c8cba7f796/volumes/kubernetes.io~projected/kube-api-access-gprnc DeviceMajor:0 DeviceMinor:257 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c8d0e1f0-620a-46c0-8234-628fb8acb572/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:323 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-365 DeviceMajor:0 DeviceMinor:365 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/92470d7928996059c02f744bd144c8e341b169b6fc78e28dcb0746dd804aee25/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/50782c2accb92735384b1ec75130c51e641b2414a95d55743e69bc02be3519ca/userdata/shm DeviceMajor:0 DeviceMinor:317 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c8d0e1f0-620a-46c0-8234-628fb8acb572/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:324 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0a478aca183c6cf MacAddress:e2:78:80:fe:6b:d2 Speed:10000 Mtu:8901} {Name:397510d0f15462f MacAddress:ce:5a:40:56:b2:95 Speed:10000 Mtu:8901} {Name:409e18ce6451a89 MacAddress:4a:ef:26:c1:44:3a Speed:10000 Mtu:8901} {Name:41dc65a16bae60a MacAddress:96:65:e7:d1:8f:82 Speed:10000 Mtu:8901} {Name:50782c2accb9273 MacAddress:2e:a8:27:5b:55:d2 Speed:10000 Mtu:8901} {Name:707fc65a7121853 MacAddress:e2:ad:74:38:d7:b9 Speed:10000 Mtu:8901} {Name:b665f50eb4a136f MacAddress:0a:ef:5a:3c:27:d4 Speed:10000 Mtu:8901} {Name:bbef4aa45643805 MacAddress:1a:0d:ee:6f:46:3b Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d1:05:7d:bb Speed:0 Mtu:9001} {Name:br-int MacAddress:82:d3:8d:81:fc:41 Speed:0 Mtu:8901} {Name:cca062ee4fe9f40 MacAddress:6e:18:82:69:a0:3d Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d1:05:7d:bb Speed:0 Mtu:9001} {Name:fe4d0060951c665 MacAddress:52:47:28:0b:15:01 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:de:61:86:38:5b:5e Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:a2:09:34:f6:e8:dc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 23:59:23.425565 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.423947 5274 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 23:59:23.425565 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.424019 5274 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 23:59:23.425565 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.424305 5274 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 23:59:23.425565 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.424328 5274 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-70.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 23:59:23.425565 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.424458 5274 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 23:59:23.425565 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.424467 5274 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 23:59:23.425565 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.424478 5274 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 23:59:23.425565 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.424490 5274 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 23:59:23.425565 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.424589 5274 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 23:59:23.425565 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.424746 5274 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 23:59:23.425565 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.424867 5274 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 23:59:23.425565 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.424901 5274 kubelet.go:493] "Attempting to sync node with API server" Oct 30 23:59:23.425565 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.424911 5274 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 23:59:23.425565 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.424921 5274 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 23:59:23.425565 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.424931 5274 kubelet.go:405] "Adding apiserver pod source" Oct 30 23:59:23.425565 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.424939 5274 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 23:59:23.426531 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.425630 5274 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 23:59:23.426531 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.425647 5274 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 23:59:23.426531 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.426149 5274 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 23:59:23.428577 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.428559 5274 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 23:59:23.428577 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.428573 5274 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 23:59:23.428699 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.428666 5274 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 23:59:23.428867 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.428856 5274 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 23:59:23.428898 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.428886 5274 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 23:59:23.429003 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.428990 5274 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 23:59:23.429003 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.429004 5274 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 23:59:23.429106 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.429011 5274 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 23:59:23.429106 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.429017 5274 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 23:59:23.429106 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.429023 5274 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 23:59:23.429106 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.429029 5274 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 23:59:23.429106 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.429034 5274 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 23:59:23.429106 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.429040 5274 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 23:59:23.429106 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.429047 5274 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 23:59:23.429106 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.429052 5274 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 23:59:23.429106 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.429070 5274 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 23:59:23.429106 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.429080 5274 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 23:59:23.429106 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.429111 5274 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 23:59:23.429618 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.429117 5274 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 23:59:23.432916 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.432898 5274 server.go:1263] "Started kubelet" Oct 30 23:59:23.433045 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.432992 5274 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 23:59:23.433045 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.433003 5274 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 23:59:23.433143 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.433076 5274 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 23:59:23.438181 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.434297 5274 server.go:317] "Adding debug handlers to kubelet server" Oct 30 23:59:23.438181 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.437630 5274 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 23:59:23.433643 ip-10-0-129-70 systemd[1]: Started Kubernetes Kubelet. Oct 30 23:59:23.438689 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.438672 5274 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 23:59:23.440357 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.438710 5274 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 23:59:23.440357 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.438754 5274 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 23:59:23.440357 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.438772 5274 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 23:59:23.440357 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.438785 5274 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:53:14 +0000 UTC" deadline="2028-07-10 01:25:28.167430326 +0000 UTC" Oct 30 23:59:23.440357 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.438807 5274 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14833h26m4.728626848s" Oct 30 23:59:23.440357 ip-10-0-129-70 kubenswrapper[5274]: E1030 23:59:23.438832 5274 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-70.us-east-2.compute.internal\" not found" Oct 30 23:59:23.440357 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.438913 5274 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 23:59:23.441619 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.441580 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a7a184e-d0ef-4c2a-ab06-129e37b7e22e" volumeName="kubernetes.io/secret/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-secret-metrics-server-tls" seLinuxMountContext="" Oct 30 23:59:23.441619 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.441617 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ad2242c-28d7-4495-9bac-f6c8cba7f796" volumeName="kubernetes.io/secret/1ad2242c-28d7-4495-9bac-f6c8cba7f796-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 23:59:23.441788 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.441630 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25e99515-222f-4346-9fe5-5b7b3b213ee8" volumeName="kubernetes.io/projected/25e99515-222f-4346-9fe5-5b7b3b213ee8-kube-api-access-dpxpg" seLinuxMountContext="" Oct 30 23:59:23.441788 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.441643 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="409b7296-1d35-44e9-be1a-3459f0dbaad9" volumeName="kubernetes.io/secret/409b7296-1d35-44e9-be1a-3459f0dbaad9-agent-certs" seLinuxMountContext="" Oct 30 23:59:23.441788 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.441654 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5405bfc2-cfc0-4ea6-80bc-911e117f1843" volumeName="kubernetes.io/empty-dir/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-tuned" seLinuxMountContext="" Oct 30 23:59:23.441788 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.441666 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5405bfc2-cfc0-4ea6-80bc-911e117f1843" volumeName="kubernetes.io/projected/5405bfc2-cfc0-4ea6-80bc-911e117f1843-kube-api-access-mcvd6" seLinuxMountContext="" Oct 30 23:59:23.441788 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.441677 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0be3213-788c-45b4-8fbd-37e744a14f17" volumeName="kubernetes.io/projected/a0be3213-788c-45b4-8fbd-37e744a14f17-kube-api-access-hlqqp" seLinuxMountContext="" Oct 30 23:59:23.441788 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.441688 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ab2f199-64ea-4071-b8cd-6b9861d31179" volumeName="kubernetes.io/secret/5ab2f199-64ea-4071-b8cd-6b9861d31179-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 23:59:23.441788 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.441702 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b099593-ded8-475c-b5ef-a91bbd9aeb1a" volumeName="kubernetes.io/projected/8b099593-ded8-475c-b5ef-a91bbd9aeb1a-kube-api-access-gnlg7" seLinuxMountContext="" Oct 30 23:59:23.441788 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.441715 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8089437-f546-441e-b121-6e79878d2608" volumeName="kubernetes.io/configmap/d8089437-f546-441e-b121-6e79878d2608-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 23:59:23.441788 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.441726 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50c15c32-e5cd-426c-9e7d-c123f8480653" volumeName="kubernetes.io/projected/50c15c32-e5cd-426c-9e7d-c123f8480653-kube-api-access-gd4j2" seLinuxMountContext="" Oct 30 23:59:23.441788 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.441737 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b099593-ded8-475c-b5ef-a91bbd9aeb1a" volumeName="kubernetes.io/secret/8b099593-ded8-475c-b5ef-a91bbd9aeb1a-metrics-tls" seLinuxMountContext="" Oct 30 23:59:23.441788 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.441749 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8d0e1f0-620a-46c0-8234-628fb8acb572" volumeName="kubernetes.io/configmap/c8d0e1f0-620a-46c0-8234-628fb8acb572-metrics-client-ca" seLinuxMountContext="" Oct 30 23:59:23.441788 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.441763 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8d0e1f0-620a-46c0-8234-628fb8acb572" volumeName="kubernetes.io/secret/c8d0e1f0-620a-46c0-8234-628fb8acb572-telemeter-client-tls" seLinuxMountContext="" Oct 30 23:59:23.441788 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.441778 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5405bfc2-cfc0-4ea6-80bc-911e117f1843" volumeName="kubernetes.io/empty-dir/5405bfc2-cfc0-4ea6-80bc-911e117f1843-tmp" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.441803 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ab2f199-64ea-4071-b8cd-6b9861d31179" volumeName="kubernetes.io/configmap/5ab2f199-64ea-4071-b8cd-6b9861d31179-metrics-client-ca" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.441815 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8d0e1f0-620a-46c0-8234-628fb8acb572" volumeName="kubernetes.io/secret/c8d0e1f0-620a-46c0-8234-628fb8acb572-secret-telemeter-client" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.441829 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8089437-f546-441e-b121-6e79878d2608" volumeName="kubernetes.io/configmap/d8089437-f546-441e-b121-6e79878d2608-cni-binary-copy" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.441843 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ad2242c-28d7-4495-9bac-f6c8cba7f796" volumeName="kubernetes.io/projected/1ad2242c-28d7-4495-9bac-f6c8cba7f796-kube-api-access-gprnc" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.441854 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8089437-f546-441e-b121-6e79878d2608" volumeName="kubernetes.io/projected/d8089437-f546-441e-b121-6e79878d2608-kube-api-access-d8wrl" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.441866 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9" volumeName="kubernetes.io/configmap/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.441880 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0adc098d-57a4-490e-89b7-b1b271a1c087" volumeName="kubernetes.io/secret/0adc098d-57a4-490e-89b7-b1b271a1c087-metrics-certs" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.441894 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63e868ff-3f36-4542-9e1d-9688ce8c37f1" volumeName="kubernetes.io/projected/63e868ff-3f36-4542-9e1d-9688ce8c37f1-kube-api-access-fl2s5" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.441907 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99b4d92d-2003-4708-99da-a5f12cbb10e2" volumeName="kubernetes.io/secret/99b4d92d-2003-4708-99da-a5f12cbb10e2-original-pull-secret" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.441919 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ad2242c-28d7-4495-9bac-f6c8cba7f796" volumeName="kubernetes.io/configmap/1ad2242c-28d7-4495-9bac-f6c8cba7f796-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.441931 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25e99515-222f-4346-9fe5-5b7b3b213ee8" volumeName="kubernetes.io/secret/25e99515-222f-4346-9fe5-5b7b3b213ee8-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.441943 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cb65abb-ba0e-4075-9a0a-455e1b64b820" volumeName="kubernetes.io/secret/3cb65abb-ba0e-4075-9a0a-455e1b64b820-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.441973 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7815306-7a99-4fc5-8226-b34916ceaaa2" volumeName="kubernetes.io/empty-dir/d7815306-7a99-4fc5-8226-b34916ceaaa2-tmp-dir" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.441997 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9" volumeName="kubernetes.io/configmap/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-metrics-client-ca" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442010 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9" volumeName="kubernetes.io/projected/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-kube-api-access-85v2m" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442022 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0adc098d-57a4-490e-89b7-b1b271a1c087" volumeName="kubernetes.io/projected/0adc098d-57a4-490e-89b7-b1b271a1c087-kube-api-access-djj7v" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442034 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ab2f199-64ea-4071-b8cd-6b9861d31179" volumeName="kubernetes.io/secret/5ab2f199-64ea-4071-b8cd-6b9861d31179-openshift-state-metrics-tls" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442046 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8d0e1f0-620a-46c0-8234-628fb8acb572" volumeName="kubernetes.io/configmap/c8d0e1f0-620a-46c0-8234-628fb8acb572-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442066 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ad2242c-28d7-4495-9bac-f6c8cba7f796" volumeName="kubernetes.io/configmap/1ad2242c-28d7-4495-9bac-f6c8cba7f796-metrics-client-ca" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442078 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25e99515-222f-4346-9fe5-5b7b3b213ee8" volumeName="kubernetes.io/configmap/25e99515-222f-4346-9fe5-5b7b3b213ee8-ovnkube-config" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442089 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25e99515-222f-4346-9fe5-5b7b3b213ee8" volumeName="kubernetes.io/configmap/25e99515-222f-4346-9fe5-5b7b3b213ee8-env-overrides" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442101 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25e99515-222f-4346-9fe5-5b7b3b213ee8" volumeName="kubernetes.io/configmap/25e99515-222f-4346-9fe5-5b7b3b213ee8-ovnkube-script-lib" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442113 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b099593-ded8-475c-b5ef-a91bbd9aeb1a" volumeName="kubernetes.io/configmap/8b099593-ded8-475c-b5ef-a91bbd9aeb1a-config-volume" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442125 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8d0e1f0-620a-46c0-8234-628fb8acb572" volumeName="kubernetes.io/configmap/c8d0e1f0-620a-46c0-8234-628fb8acb572-telemeter-trusted-ca-bundle" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442138 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8d0e1f0-620a-46c0-8234-628fb8acb572" volumeName="kubernetes.io/projected/c8d0e1f0-620a-46c0-8234-628fb8acb572-kube-api-access-znq6p" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442149 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9" volumeName="kubernetes.io/empty-dir/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-node-exporter-textfile" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442160 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b099593-ded8-475c-b5ef-a91bbd9aeb1a" volumeName="kubernetes.io/empty-dir/8b099593-ded8-475c-b5ef-a91bbd9aeb1a-tmp-dir" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442172 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0be3213-788c-45b4-8fbd-37e744a14f17" volumeName="kubernetes.io/configmap/a0be3213-788c-45b4-8fbd-37e744a14f17-multus-daemon-config" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442185 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8089437-f546-441e-b121-6e79878d2608" volumeName="kubernetes.io/configmap/d8089437-f546-441e-b121-6e79878d2608-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442196 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9" volumeName="kubernetes.io/secret/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-node-exporter-tls" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442220 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18698607-3254-4ec9-b3bb-bbe59c6c744f" volumeName="kubernetes.io/secret/18698607-3254-4ec9-b3bb-bbe59c6c744f-cert" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442233 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f653a44-907a-417e-a1da-9b1bb0ee7283" volumeName="kubernetes.io/projected/1f653a44-907a-417e-a1da-9b1bb0ee7283-kube-api-access-r8tds" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442271 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cb65abb-ba0e-4075-9a0a-455e1b64b820" volumeName="kubernetes.io/configmap/3cb65abb-ba0e-4075-9a0a-455e1b64b820-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442287 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a7a184e-d0ef-4c2a-ab06-129e37b7e22e" volumeName="kubernetes.io/empty-dir/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-audit-log" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442301 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ad2242c-28d7-4495-9bac-f6c8cba7f796" volumeName="kubernetes.io/empty-dir/1ad2242c-28d7-4495-9bac-f6c8cba7f796-volume-directive-shadow" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442314 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ad2242c-28d7-4495-9bac-f6c8cba7f796" volumeName="kubernetes.io/secret/1ad2242c-28d7-4495-9bac-f6c8cba7f796-kube-state-metrics-tls" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442328 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="409b7296-1d35-44e9-be1a-3459f0dbaad9" volumeName="kubernetes.io/configmap/409b7296-1d35-44e9-be1a-3459f0dbaad9-konnectivity-ca" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442341 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ab2f199-64ea-4071-b8cd-6b9861d31179" volumeName="kubernetes.io/projected/5ab2f199-64ea-4071-b8cd-6b9861d31179-kube-api-access-qfgxg" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442353 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8d0e1f0-620a-46c0-8234-628fb8acb572" volumeName="kubernetes.io/secret/c8d0e1f0-620a-46c0-8234-628fb8acb572-federate-client-tls" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442365 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8d0e1f0-620a-46c0-8234-628fb8acb572" volumeName="kubernetes.io/secret/c8d0e1f0-620a-46c0-8234-628fb8acb572-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442378 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a7a184e-d0ef-4c2a-ab06-129e37b7e22e" volumeName="kubernetes.io/secret/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-client-ca-bundle" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442399 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18698607-3254-4ec9-b3bb-bbe59c6c744f" volumeName="kubernetes.io/projected/18698607-3254-4ec9-b3bb-bbe59c6c744f-kube-api-access-cqkgh" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442413 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18acd0a8-988e-4168-84e7-0b25dc7b32d4" volumeName="kubernetes.io/configmap/18acd0a8-988e-4168-84e7-0b25dc7b32d4-serviceca" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442426 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f653a44-907a-417e-a1da-9b1bb0ee7283" volumeName="kubernetes.io/configmap/1f653a44-907a-417e-a1da-9b1bb0ee7283-iptables-alerter-script" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442439 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cb65abb-ba0e-4075-9a0a-455e1b64b820" volumeName="kubernetes.io/projected/3cb65abb-ba0e-4075-9a0a-455e1b64b820-kube-api-access-wsblw" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442451 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a7a184e-d0ef-4c2a-ab06-129e37b7e22e" volumeName="kubernetes.io/configmap/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442464 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a7a184e-d0ef-4c2a-ab06-129e37b7e22e" volumeName="kubernetes.io/projected/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-kube-api-access-5fr52" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442477 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a7a184e-d0ef-4c2a-ab06-129e37b7e22e" volumeName="kubernetes.io/secret/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442489 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18acd0a8-988e-4168-84e7-0b25dc7b32d4" volumeName="kubernetes.io/projected/18acd0a8-988e-4168-84e7-0b25dc7b32d4-kube-api-access-pmcgf" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442502 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0be3213-788c-45b4-8fbd-37e744a14f17" volumeName="kubernetes.io/configmap/a0be3213-788c-45b4-8fbd-37e744a14f17-cni-binary-copy" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442514 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7815306-7a99-4fc5-8226-b34916ceaaa2" volumeName="kubernetes.io/projected/d7815306-7a99-4fc5-8226-b34916ceaaa2-kube-api-access-2kjxh" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442527 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a7a184e-d0ef-4c2a-ab06-129e37b7e22e" volumeName="kubernetes.io/configmap/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-metrics-server-audit-profiles" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442538 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cb65abb-ba0e-4075-9a0a-455e1b64b820" volumeName="kubernetes.io/empty-dir/3cb65abb-ba0e-4075-9a0a-455e1b64b820-data-volume" seLinuxMountContext="" Oct 30 23:59:23.442494 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442550 5274 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9" volumeName="kubernetes.io/secret/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 23:59:23.444615 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442561 5274 reconstruct.go:97] "Volume reconstruction finished" Oct 30 23:59:23.444615 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.442568 5274 reconciler.go:29] "Reconciler: start to sync state" Oct 30 23:59:23.444615 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.443911 5274 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:59:23.445522 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.445502 5274 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:59:23.447561 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.447543 5274 factory.go:55] Registering systemd factory Oct 30 23:59:23.447561 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.447564 5274 factory.go:223] Registration of the systemd container factory successfully Oct 30 23:59:23.447810 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.447794 5274 factory.go:153] Registering CRI-O factory Oct 30 23:59:23.447854 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.447816 5274 factory.go:223] Registration of the crio container factory successfully Oct 30 23:59:23.447903 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.447876 5274 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 23:59:23.447942 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.447903 5274 factory.go:103] Registering Raw factory Oct 30 23:59:23.447942 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.447918 5274 manager.go:1199] Started watching for new ooms in manager Oct 30 23:59:23.448184 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.448151 5274 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:59:23.448295 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.448281 5274 manager.go:319] Starting recovery of all containers Oct 30 23:59:23.448458 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.448437 5274 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 23:59:23.473223 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.472989 5274 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 23:59:23.474622 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.474592 5274 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 23:59:23.474758 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.474696 5274 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 23:59:23.474758 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.474722 5274 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 23:59:23.474852 ip-10-0-129-70 kubenswrapper[5274]: E1030 23:59:23.474767 5274 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 23:59:23.478613 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.478446 5274 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:59:23.501869 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.501830 5274 generic.go:356] "Generic (PLEG): container finished" podID="c4b01b095f68d38f7d23560b244b5e01" containerID="d2013d77d668064652b566a512f5446c43c712219e78c1d3847a2acea0b1b2cc" exitCode=0 Oct 30 23:59:23.524544 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.524503 5274 generic.go:356] "Generic (PLEG): container finished" podID="de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9" containerID="a434bd5bf75f6333c98ef5bf7ba66491eeace7c6c464c27f7eba21c64661b912" exitCode=0 Oct 30 23:59:23.538782 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.538748 5274 generic.go:356] "Generic (PLEG): container finished" podID="d8089437-f546-441e-b121-6e79878d2608" containerID="814815e3349f2a8643c4870bb191fdf48911ad2fe41f586127b1302fab55bf2a" exitCode=0 Oct 30 23:59:23.538782 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.538778 5274 generic.go:356] "Generic (PLEG): container finished" podID="d8089437-f546-441e-b121-6e79878d2608" containerID="cf2969e21000e9780f10eea9d68d869842f5e96f9bd0a43e5b22fe83d49a2b98" exitCode=0 Oct 30 23:59:23.538782 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.538786 5274 generic.go:356] "Generic (PLEG): container finished" podID="d8089437-f546-441e-b121-6e79878d2608" containerID="6daef47f6f360f0ba9f06ca03e6ca549bcbd61c14a8004c7aa81244c036ab7e7" exitCode=0 Oct 30 23:59:23.538782 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.538793 5274 generic.go:356] "Generic (PLEG): container finished" podID="d8089437-f546-441e-b121-6e79878d2608" containerID="912e17c20f5cc26bd9e7e77ddfef14466a60d2601e36aee8401c92b5c26bfc4f" exitCode=0 Oct 30 23:59:23.539055 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.538798 5274 generic.go:356] "Generic (PLEG): container finished" podID="d8089437-f546-441e-b121-6e79878d2608" containerID="0aa944992685d16c0edc63b91222b665fd4fd75181c5c310834b3b9f8685c224" exitCode=0 Oct 30 23:59:23.539055 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.538803 5274 generic.go:356] "Generic (PLEG): container finished" podID="d8089437-f546-441e-b121-6e79878d2608" containerID="fd14f9bd44e7d8b2299bc7aaf2e6102a6ec274b31046a99b3ceb1d4cb24dc718" exitCode=0 Oct 30 23:59:23.551080 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.551047 5274 manager.go:324] Recovery completed Oct 30 23:59:23.571855 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.571774 5274 cpu_manager.go:226] "Starting" policy="none" Oct 30 23:59:23.571855 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.571797 5274 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 23:59:23.571855 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.571825 5274 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 23:59:23.572062 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.571977 5274 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 23:59:23.572062 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.571989 5274 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 23:59:23.572062 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.572007 5274 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 23:59:23.572062 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.572013 5274 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 23:59:23.572062 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.572018 5274 policy_none.go:50] "Start" Oct 30 23:59:23.572062 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.572027 5274 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 23:59:23.572062 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.572036 5274 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 23:59:23.573349 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.573331 5274 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 23:59:23.573349 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.573349 5274 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 23:59:23.573435 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.573356 5274 policy_none.go:44] "Start" Oct 30 23:59:23.574876 ip-10-0-129-70 kubenswrapper[5274]: E1030 23:59:23.574855 5274 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 23:59:23.577902 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.577873 5274 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 23:59:23.583567 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.583544 5274 manager.go:341] "Starting Device Plugin manager" Oct 30 23:59:23.584072 ip-10-0-129-70 kubenswrapper[5274]: E1030 23:59:23.583580 5274 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 23:59:23.584164 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.584075 5274 server.go:87] "Starting device plugin registration server" Oct 30 23:59:23.584408 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.584393 5274 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 23:59:23.584490 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.584409 5274 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 23:59:23.584537 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.584511 5274 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 23:59:23.584677 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.584637 5274 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 23:59:23.584783 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.584685 5274 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 23:59:23.584783 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.584693 5274 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 23:59:23.584888 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.584746 5274 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T23:59:23.584672964Z","UUID":"1f3c1605-0c3a-4c21-9eee-82aee853b99c","Handler":null,"Name":"","Endpoint":""} Oct 30 23:59:23.586117 ip-10-0-129-70 kubenswrapper[5274]: E1030 23:59:23.586097 5274 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 23:59:23.586461 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.586441 5274 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 23:59:23.586582 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.586470 5274 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 23:59:23.685500 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.685467 5274 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:59:23.687641 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.687622 5274 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-70.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:59:23.687641 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.687652 5274 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-70.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:59:23.687850 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.687662 5274 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-70.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:59:23.687850 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.687700 5274 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:59:23.700238 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.700205 5274 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:59:23.700380 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.700299 5274 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:59:23.776013 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.775929 5274 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-70.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal"] Oct 30 23:59:23.776580 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.776540 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal" event={"ID":"c4b01b095f68d38f7d23560b244b5e01","Type":"ContainerStarted","Data":"0d873c115b99713928439afe81106a9c4306a9334bad5b00fc54ad3c142c3519"} Oct 30 23:59:23.776720 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.776588 5274 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:59:23.776720 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.776588 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal" event={"ID":"c4b01b095f68d38f7d23560b244b5e01","Type":"ContainerDied","Data":"d2013d77d668064652b566a512f5446c43c712219e78c1d3847a2acea0b1b2cc"} Oct 30 23:59:23.776812 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.776718 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal" event={"ID":"c4b01b095f68d38f7d23560b244b5e01","Type":"ContainerStarted","Data":"a5b7fb8850839f327a21ed0ca1950d02333706f22908cae2ae7631d85e53f37b"} Oct 30 23:59:23.776812 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.776787 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-70.us-east-2.compute.internal" event={"ID":"2e9a3bb1bd7fbf1e3c59a1731918cc24","Type":"ContainerStarted","Data":"41b94ee11f573ef72e0c10cb0486a9b0b00e8801cebce2f917c72407c6d3c8ed"} Oct 30 23:59:23.776812 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.776807 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-70.us-east-2.compute.internal" event={"ID":"2e9a3bb1bd7fbf1e3c59a1731918cc24","Type":"ContainerStarted","Data":"34539fa69c119c636d1acc43e480f7606c57d5d847b5c124491b8ea368c84256"} Oct 30 23:59:23.776929 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.776901 5274 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:59:23.788948 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.788927 5274 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:59:23.789034 ip-10-0-129-70 kubenswrapper[5274]: E1030 23:59:23.789000 5274 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-70.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:59:23.789803 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.789787 5274 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:59:23.789842 ip-10-0-129-70 kubenswrapper[5274]: E1030 23:59:23.789829 5274 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:59:23.850731 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.850663 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c4b01b095f68d38f7d23560b244b5e01-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal\" (UID: \"c4b01b095f68d38f7d23560b244b5e01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:59:23.850731 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.850699 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4b01b095f68d38f7d23560b244b5e01-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal\" (UID: \"c4b01b095f68d38f7d23560b244b5e01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:59:23.850731 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.850722 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2e9a3bb1bd7fbf1e3c59a1731918cc24-config\") pod \"kube-apiserver-proxy-ip-10-0-129-70.us-east-2.compute.internal\" (UID: \"2e9a3bb1bd7fbf1e3c59a1731918cc24\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:59:23.951408 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.951373 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c4b01b095f68d38f7d23560b244b5e01-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal\" (UID: \"c4b01b095f68d38f7d23560b244b5e01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:59:23.951558 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.951402 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c4b01b095f68d38f7d23560b244b5e01-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal\" (UID: \"c4b01b095f68d38f7d23560b244b5e01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:59:23.951558 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.951445 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4b01b095f68d38f7d23560b244b5e01-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal\" (UID: \"c4b01b095f68d38f7d23560b244b5e01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:59:23.951558 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.951465 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2e9a3bb1bd7fbf1e3c59a1731918cc24-config\") pod \"kube-apiserver-proxy-ip-10-0-129-70.us-east-2.compute.internal\" (UID: \"2e9a3bb1bd7fbf1e3c59a1731918cc24\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:59:23.951558 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.951494 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2e9a3bb1bd7fbf1e3c59a1731918cc24-config\") pod \"kube-apiserver-proxy-ip-10-0-129-70.us-east-2.compute.internal\" (UID: \"2e9a3bb1bd7fbf1e3c59a1731918cc24\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:59:23.951558 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:23.951518 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4b01b095f68d38f7d23560b244b5e01-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal\" (UID: \"c4b01b095f68d38f7d23560b244b5e01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:59:24.425292 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.425248 5274 apiserver.go:52] "Watching apiserver" Oct 30 23:59:24.443729 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.443694 5274 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 23:59:24.444265 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.444240 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-7zxsz","openshift-ingress-canary/ingress-canary-djtcv","openshift-multus/network-metrics-daemon-cjqfw","openshift-network-operator/iptables-alerter-zlgls","kube-system/global-pull-secret-syncer-6l8ht","kube-system/konnectivity-agent-x962k","openshift-dns/node-resolver-6hq7b","openshift-network-diagnostics/network-check-target-bd4bx","openshift-dns/dns-default-g5p2k","openshift-image-registry/node-ca-vnk6j","openshift-monitoring/node-exporter-bpczc","openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w","openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp","kube-system/kube-apiserver-proxy-ip-10-0-129-70.us-east-2.compute.internal","openshift-monitoring/telemeter-client-5dc5757857-pp2xd","openshift-monitoring/openshift-state-metrics-7bd7489957-tt54l","openshift-insights/insights-runtime-extractor-84g8j","openshift-multus/multus-additional-cni-plugins-tc87d","openshift-multus/multus-9j82q","openshift-cluster-node-tuning-operator/tuned-f64c8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal"] Oct 30 23:59:24.452351 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.452320 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:59:24.455860 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.455827 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 23:59:24.456031 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.455930 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 23:59:24.456031 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.455948 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 23:59:24.456031 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.455977 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 23:59:24.456211 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.456039 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 23:59:24.456211 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.456051 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 23:59:24.456211 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.456059 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 23:59:24.456211 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.456106 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-72z8g\"" Oct 30 23:59:24.456211 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.456121 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 23:59:24.456211 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.456128 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 23:59:24.456211 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.456136 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rqcpl\"" Oct 30 23:59:24.456211 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.456105 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 23:59:24.456211 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.456159 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 23:59:24.456211 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.456165 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 23:59:24.456211 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.456129 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 23:59:24.456211 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.456176 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 23:59:24.456211 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.456192 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 23:59:24.456211 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.456167 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 23:59:24.456211 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.456209 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 23:59:24.456211 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.456217 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 23:59:24.456626 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.456218 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 23:59:24.456626 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.456250 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 23:59:24.456768 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.456745 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 23:59:24.456768 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.456754 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pxpmk\"" Oct 30 23:59:24.457102 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.457086 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hrv6t\"" Oct 30 23:59:24.457160 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.457090 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 23:59:24.457160 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.457113 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dtg7z\"" Oct 30 23:59:24.457236 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.457191 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-fl8mx\"" Oct 30 23:59:24.457236 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.457231 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xfgqn\"" Oct 30 23:59:24.457303 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.457231 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-6ff8s\"" Oct 30 23:59:24.457303 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.457261 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 23:59:24.458367 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.458321 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 23:59:24.458367 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.458335 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 23:59:24.458569 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.458373 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mmfft\"" Oct 30 23:59:24.458569 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.458388 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-g645g\"" Oct 30 23:59:24.458569 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.458481 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 23:59:24.458569 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.458545 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-r8kbq\"" Oct 30 23:59:24.459665 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.459646 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-mt26w\"" Oct 30 23:59:24.459756 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.459691 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 23:59:24.459756 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.459725 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 23:59:24.459756 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.459743 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-x2vzx\"" Oct 30 23:59:24.459914 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.459829 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 23:59:24.459985 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.459917 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9ccbg\"" Oct 30 23:59:24.461000 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.460981 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 23:59:24.461180 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.461163 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 23:59:24.461258 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.461192 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 23:59:24.461258 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.461236 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 23:59:24.461331 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.461294 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 23:59:24.462315 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.462258 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 23:59:24.462315 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.462262 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 23:59:24.462315 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.462256 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 23:59:24.462315 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.462267 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xcdsb\"" Oct 30 23:59:24.462315 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.462312 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 23:59:24.462315 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.462316 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 23:59:24.462611 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.462420 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 23:59:24.462611 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.462432 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 23:59:24.462611 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.462444 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 23:59:24.463542 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.463521 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l7b2l\"" Oct 30 23:59:24.463639 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.463544 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 23:59:24.463639 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.463549 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 23:59:24.463639 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.463582 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 23:59:24.463639 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.463592 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 23:59:24.463639 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.463597 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 23:59:24.463639 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.463602 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lxfk8\"" Oct 30 23:59:24.463639 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.463588 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:59:24.463639 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.463602 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 23:59:24.463888 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.463644 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-splp4\"" Oct 30 23:59:24.463888 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.463657 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 23:59:24.464867 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.464842 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-vmg6c\"" Oct 30 23:59:24.464943 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.464883 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 23:59:24.464943 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.464893 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-b7saddsnpolru\"" Oct 30 23:59:24.464943 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.464902 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 23:59:24.464943 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.464909 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 23:59:24.464943 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.464903 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 23:59:24.466068 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.466055 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 23:59:24.466158 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.466144 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-plh7c\"" Oct 30 23:59:24.466201 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.466159 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 23:59:24.467130 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.467116 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 23:59:24.467300 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.467285 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 23:59:24.467346 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.467286 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 23:59:24.471166 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.471143 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 23:59:24.541108 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.541083 5274 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 23:59:24.542577 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.542560 5274 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:59:24.542674 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.542661 5274 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:59:24.553572 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.553540 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-tuned\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:24.553572 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.553571 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-run-systemd\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.553751 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.553589 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/25e99515-222f-4346-9fe5-5b7b3b213ee8-ovnkube-script-lib\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.553751 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.553607 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-host-var-lib-cni-multus\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.553751 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.553651 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ab2f199-64ea-4071-b8cd-6b9861d31179-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-tt54l\" (UID: \"5ab2f199-64ea-4071-b8cd-6b9861d31179\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tt54l" Oct 30 23:59:24.553751 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.553710 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-tuned\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:24.553888 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.553753 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-host\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:24.553888 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.553790 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-cnibin\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.553888 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.553861 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-systemd-units\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.554002 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.553889 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ab2f199-64ea-4071-b8cd-6b9861d31179-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-tt54l\" (UID: \"5ab2f199-64ea-4071-b8cd-6b9861d31179\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tt54l" Oct 30 23:59:24.554002 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.553916 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/25e99515-222f-4346-9fe5-5b7b3b213ee8-ovnkube-script-lib\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.554002 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.553917 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b099593-ded8-475c-b5ef-a91bbd9aeb1a-tmp-dir\") pod \"dns-default-g5p2k\" (UID: \"8b099593-ded8-475c-b5ef-a91bbd9aeb1a\") " pod="openshift-dns/dns-default-g5p2k" Oct 30 23:59:24.554002 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.553950 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c8d0e1f0-620a-46c0-8234-628fb8acb572-federate-client-tls\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:24.554130 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.554019 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8d0e1f0-620a-46c0-8234-628fb8acb572-metrics-client-ca\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:24.554130 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.554032 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b099593-ded8-475c-b5ef-a91bbd9aeb1a-tmp-dir\") pod \"dns-default-g5p2k\" (UID: \"8b099593-ded8-475c-b5ef-a91bbd9aeb1a\") " pod="openshift-dns/dns-default-g5p2k" Oct 30 23:59:24.554130 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.554060 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-cni-netd\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.554130 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.554077 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-multus-cni-dir\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.554130 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.554095 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8d0e1f0-620a-46c0-8234-628fb8acb572-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:24.554130 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.554112 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-sysconfig\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:24.554130 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.554126 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-run\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:24.554384 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.554158 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85v2m\" (UniqueName: \"kubernetes.io/projected/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-kube-api-access-85v2m\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:24.554384 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.554190 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-host-run-multus-certs\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.554384 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.554203 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c8d0e1f0-620a-46c0-8234-628fb8acb572-federate-client-tls\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:24.554384 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.554230 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8d0e1f0-620a-46c0-8234-628fb8acb572-metrics-client-ca\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:24.554384 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.554268 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlqqp\" (UniqueName: \"kubernetes.io/projected/a0be3213-788c-45b4-8fbd-37e744a14f17-kube-api-access-hlqqp\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.554384 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.554317 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7815306-7a99-4fc5-8226-b34916ceaaa2-tmp-dir\") pod \"node-resolver-6hq7b\" (UID: \"d7815306-7a99-4fc5-8226-b34916ceaaa2\") " pod="openshift-dns/node-resolver-6hq7b" Oct 30 23:59:24.554384 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.554315 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8d0e1f0-620a-46c0-8234-628fb8acb572-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:24.554384 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.554365 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7815306-7a99-4fc5-8226-b34916ceaaa2-tmp-dir\") pod \"node-resolver-6hq7b\" (UID: \"d7815306-7a99-4fc5-8226-b34916ceaaa2\") " pod="openshift-dns/node-resolver-6hq7b" Oct 30 23:59:24.554644 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.554396 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8089437-f546-441e-b121-6e79878d2608-cnibin\") pod \"multus-additional-cni-plugins-tc87d\" (UID: \"d8089437-f546-441e-b121-6e79878d2608\") " pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:59:24.554644 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.554418 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-node-exporter-tls\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:24.554644 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.554434 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-host-var-lib-kubelet\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.554644 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.554482 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcvd6\" (UniqueName: \"kubernetes.io/projected/5405bfc2-cfc0-4ea6-80bc-911e117f1843-kube-api-access-mcvd6\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:24.554752 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.554655 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/18acd0a8-988e-4168-84e7-0b25dc7b32d4-host\") pod \"node-ca-vnk6j\" (UID: \"18acd0a8-988e-4168-84e7-0b25dc7b32d4\") " pod="openshift-image-registry/node-ca-vnk6j" Oct 30 23:59:24.554752 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.554665 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-node-exporter-tls\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:24.554752 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.554721 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmcgf\" (UniqueName: \"kubernetes.io/projected/18acd0a8-988e-4168-84e7-0b25dc7b32d4-kube-api-access-pmcgf\") pod \"node-ca-vnk6j\" (UID: \"18acd0a8-988e-4168-84e7-0b25dc7b32d4\") " pod="openshift-image-registry/node-ca-vnk6j" Oct 30 23:59:24.554752 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.554741 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.554856 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.554763 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-sysctl-conf\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:24.554884 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.554859 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-etc-openvswitch\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.554912 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.554886 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a0be3213-788c-45b4-8fbd-37e744a14f17-cni-binary-copy\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.555013 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.554915 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-host-run-k8s-cni-cncf-io\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.555013 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.554939 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-metrics-server-audit-profiles\") pod \"metrics-server-6fbfb6ff44-2sxgp\" (UID: \"0a7a184e-d0ef-4c2a-ab06-129e37b7e22e\") " pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" Oct 30 23:59:24.555013 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.554978 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d8089437-f546-441e-b121-6e79878d2608-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tc87d\" (UID: \"d8089437-f546-441e-b121-6e79878d2608\") " pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:59:24.555013 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.554998 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-kubelet\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.555192 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.555014 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpxpg\" (UniqueName: \"kubernetes.io/projected/25e99515-222f-4346-9fe5-5b7b3b213ee8-kube-api-access-dpxpg\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.555192 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.555039 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-multus-conf-dir\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.555192 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.555062 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-secret-metrics-server-tls\") pod \"metrics-server-6fbfb6ff44-2sxgp\" (UID: \"0a7a184e-d0ef-4c2a-ab06-129e37b7e22e\") " pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" Oct 30 23:59:24.555192 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.555118 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a0be3213-788c-45b4-8fbd-37e744a14f17-cni-binary-copy\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.555192 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.555107 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6fbfb6ff44-2sxgp\" (UID: \"0a7a184e-d0ef-4c2a-ab06-129e37b7e22e\") " pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" Oct 30 23:59:24.555347 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.555193 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8089437-f546-441e-b121-6e79878d2608-cni-binary-copy\") pod \"multus-additional-cni-plugins-tc87d\" (UID: \"d8089437-f546-441e-b121-6e79878d2608\") " pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:59:24.555347 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.555225 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-metrics-server-audit-profiles\") pod \"metrics-server-6fbfb6ff44-2sxgp\" (UID: \"0a7a184e-d0ef-4c2a-ab06-129e37b7e22e\") " pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" Oct 30 23:59:24.555347 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.555232 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1f653a44-907a-417e-a1da-9b1bb0ee7283-iptables-alerter-script\") pod \"iptables-alerter-zlgls\" (UID: \"1f653a44-907a-417e-a1da-9b1bb0ee7283\") " pod="openshift-network-operator/iptables-alerter-zlgls" Oct 30 23:59:24.555347 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.555269 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kjxh\" (UniqueName: \"kubernetes.io/projected/d7815306-7a99-4fc5-8226-b34916ceaaa2-kube-api-access-2kjxh\") pod \"node-resolver-6hq7b\" (UID: \"d7815306-7a99-4fc5-8226-b34916ceaaa2\") " pod="openshift-dns/node-resolver-6hq7b" Oct 30 23:59:24.555347 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.555302 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ad2242c-28d7-4495-9bac-f6c8cba7f796-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-frfhc\" (UID: \"1ad2242c-28d7-4495-9bac-f6c8cba7f796\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc" Oct 30 23:59:24.555347 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.555331 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ad2242c-28d7-4495-9bac-f6c8cba7f796-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-frfhc\" (UID: \"1ad2242c-28d7-4495-9bac-f6c8cba7f796\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc" Oct 30 23:59:24.555503 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.555361 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd4j2\" (UniqueName: \"kubernetes.io/projected/50c15c32-e5cd-426c-9e7d-c123f8480653-kube-api-access-gd4j2\") pod \"network-check-target-bd4bx\" (UID: \"50c15c32-e5cd-426c-9e7d-c123f8480653\") " pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:59:24.555503 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.555391 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:24.555503 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.555422 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-cni-bin\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.555503 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.555467 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-host-run-netns\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.555503 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.555471 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8089437-f546-441e-b121-6e79878d2608-cni-binary-copy\") pod \"multus-additional-cni-plugins-tc87d\" (UID: \"d8089437-f546-441e-b121-6e79878d2608\") " pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:59:24.555668 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.555497 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gprnc\" (UniqueName: \"kubernetes.io/projected/1ad2242c-28d7-4495-9bac-f6c8cba7f796-kube-api-access-gprnc\") pod \"kube-state-metrics-5dc8d7d89b-frfhc\" (UID: \"1ad2242c-28d7-4495-9bac-f6c8cba7f796\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc" Oct 30 23:59:24.555668 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.555536 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8089437-f546-441e-b121-6e79878d2608-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tc87d\" (UID: \"d8089437-f546-441e-b121-6e79878d2608\") " pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:59:24.555668 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.555563 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-root\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:24.555668 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.555594 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-log-socket\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.555668 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.555637 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-run-ovn-kubernetes\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.555668 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.555655 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d8089437-f546-441e-b121-6e79878d2608-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tc87d\" (UID: \"d8089437-f546-441e-b121-6e79878d2608\") " pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:59:24.555947 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.555717 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnlg7\" (UniqueName: \"kubernetes.io/projected/8b099593-ded8-475c-b5ef-a91bbd9aeb1a-kube-api-access-gnlg7\") pod \"dns-default-g5p2k\" (UID: \"8b099593-ded8-475c-b5ef-a91bbd9aeb1a\") " pod="openshift-dns/dns-default-g5p2k" Oct 30 23:59:24.555947 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.555742 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/63e868ff-3f36-4542-9e1d-9688ce8c37f1-etc-selinux\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:59:24.555947 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.555745 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6fbfb6ff44-2sxgp\" (UID: \"0a7a184e-d0ef-4c2a-ab06-129e37b7e22e\") " pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" Oct 30 23:59:24.555947 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.555756 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ad2242c-28d7-4495-9bac-f6c8cba7f796-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-frfhc\" (UID: \"1ad2242c-28d7-4495-9bac-f6c8cba7f796\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc" Oct 30 23:59:24.555947 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.555769 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ab2f199-64ea-4071-b8cd-6b9861d31179-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-tt54l\" (UID: \"5ab2f199-64ea-4071-b8cd-6b9861d31179\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tt54l" Oct 30 23:59:24.555947 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.555838 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1f653a44-907a-417e-a1da-9b1bb0ee7283-iptables-alerter-script\") pod \"iptables-alerter-zlgls\" (UID: \"1f653a44-907a-417e-a1da-9b1bb0ee7283\") " pod="openshift-network-operator/iptables-alerter-zlgls" Oct 30 23:59:24.555947 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.555902 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-var-lib-openvswitch\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.555947 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.555936 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-multus-socket-dir-parent\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.556315 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.555943 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ad2242c-28d7-4495-9bac-f6c8cba7f796-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-frfhc\" (UID: \"1ad2242c-28d7-4495-9bac-f6c8cba7f796\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc" Oct 30 23:59:24.556315 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.556007 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3cb65abb-ba0e-4075-9a0a-455e1b64b820-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-84g8j\" (UID: \"3cb65abb-ba0e-4075-9a0a-455e1b64b820\") " pod="openshift-insights/insights-runtime-extractor-84g8j" Oct 30 23:59:24.556315 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.556007 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ab2f199-64ea-4071-b8cd-6b9861d31179-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-tt54l\" (UID: \"5ab2f199-64ea-4071-b8cd-6b9861d31179\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tt54l" Oct 30 23:59:24.556315 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.556029 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-secret-metrics-server-tls\") pod \"metrics-server-6fbfb6ff44-2sxgp\" (UID: \"0a7a184e-d0ef-4c2a-ab06-129e37b7e22e\") " pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" Oct 30 23:59:24.556315 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.556037 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/18acd0a8-988e-4168-84e7-0b25dc7b32d4-serviceca\") pod \"node-ca-vnk6j\" (UID: \"18acd0a8-988e-4168-84e7-0b25dc7b32d4\") " pod="openshift-image-registry/node-ca-vnk6j" Oct 30 23:59:24.556315 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.556070 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-sysctl-d\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:24.556315 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.556145 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-sys\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:24.556315 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.556177 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-metrics-client-ca\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:24.556315 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.556182 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3cb65abb-ba0e-4075-9a0a-455e1b64b820-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-84g8j\" (UID: \"3cb65abb-ba0e-4075-9a0a-455e1b64b820\") " pod="openshift-insights/insights-runtime-extractor-84g8j" Oct 30 23:59:24.556315 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.556234 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:24.556315 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.556289 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/409b7296-1d35-44e9-be1a-3459f0dbaad9-konnectivity-ca\") pod \"konnectivity-agent-x962k\" (UID: \"409b7296-1d35-44e9-be1a-3459f0dbaad9\") " pod="kube-system/konnectivity-agent-x962k" Oct 30 23:59:24.556614 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.556315 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3cb65abb-ba0e-4075-9a0a-455e1b64b820-data-volume\") pod \"insights-runtime-extractor-84g8j\" (UID: \"3cb65abb-ba0e-4075-9a0a-455e1b64b820\") " pod="openshift-insights/insights-runtime-extractor-84g8j" Oct 30 23:59:24.556614 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.556347 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fr52\" (UniqueName: \"kubernetes.io/projected/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-kube-api-access-5fr52\") pod \"metrics-server-6fbfb6ff44-2sxgp\" (UID: \"0a7a184e-d0ef-4c2a-ab06-129e37b7e22e\") " pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" Oct 30 23:59:24.556614 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.556364 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-metrics-client-ca\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:24.556614 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.556373 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8wrl\" (UniqueName: \"kubernetes.io/projected/d8089437-f546-441e-b121-6e79878d2608-kube-api-access-d8wrl\") pod \"multus-additional-cni-plugins-tc87d\" (UID: \"d8089437-f546-441e-b121-6e79878d2608\") " pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:59:24.556614 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.556426 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3cb65abb-ba0e-4075-9a0a-455e1b64b820-data-volume\") pod \"insights-runtime-extractor-84g8j\" (UID: \"3cb65abb-ba0e-4075-9a0a-455e1b64b820\") " pod="openshift-insights/insights-runtime-extractor-84g8j" Oct 30 23:59:24.556614 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.556482 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/409b7296-1d35-44e9-be1a-3459f0dbaad9-konnectivity-ca\") pod \"konnectivity-agent-x962k\" (UID: \"409b7296-1d35-44e9-be1a-3459f0dbaad9\") " pod="kube-system/konnectivity-agent-x962k" Oct 30 23:59:24.556614 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.556491 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0adc098d-57a4-490e-89b7-b1b271a1c087-metrics-certs\") pod \"network-metrics-daemon-cjqfw\" (UID: \"0adc098d-57a4-490e-89b7-b1b271a1c087\") " pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:59:24.556614 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.556535 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/63e868ff-3f36-4542-9e1d-9688ce8c37f1-registration-dir\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:59:24.556614 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.556578 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-kubernetes\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:24.556876 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.556617 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqkgh\" (UniqueName: \"kubernetes.io/projected/18698607-3254-4ec9-b3bb-bbe59c6c744f-kube-api-access-cqkgh\") pod \"ingress-canary-djtcv\" (UID: \"18698607-3254-4ec9-b3bb-bbe59c6c744f\") " pod="openshift-ingress-canary/ingress-canary-djtcv" Oct 30 23:59:24.556876 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.556659 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-node-exporter-textfile\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:24.556876 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.556686 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d7815306-7a99-4fc5-8226-b34916ceaaa2-hosts-file\") pod \"node-resolver-6hq7b\" (UID: \"d7815306-7a99-4fc5-8226-b34916ceaaa2\") " pod="openshift-dns/node-resolver-6hq7b" Oct 30 23:59:24.556876 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.556738 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/99b4d92d-2003-4708-99da-a5f12cbb10e2-kubelet-config\") pod \"global-pull-secret-syncer-6l8ht\" (UID: \"99b4d92d-2003-4708-99da-a5f12cbb10e2\") " pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:59:24.556876 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.556748 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-node-exporter-textfile\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:24.556876 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.556750 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/18acd0a8-988e-4168-84e7-0b25dc7b32d4-serviceca\") pod \"node-ca-vnk6j\" (UID: \"18acd0a8-988e-4168-84e7-0b25dc7b32d4\") " pod="openshift-image-registry/node-ca-vnk6j" Oct 30 23:59:24.556876 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.556774 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1ad2242c-28d7-4495-9bac-f6c8cba7f796-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-frfhc\" (UID: \"1ad2242c-28d7-4495-9bac-f6c8cba7f796\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc" Oct 30 23:59:24.556876 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.556795 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-node-exporter-accelerators-collector-config\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:24.556876 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.556739 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0adc098d-57a4-490e-89b7-b1b271a1c087-metrics-certs\") pod \"network-metrics-daemon-cjqfw\" (UID: \"0adc098d-57a4-490e-89b7-b1b271a1c087\") " pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:59:24.556876 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.556823 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ad2242c-28d7-4495-9bac-f6c8cba7f796-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-frfhc\" (UID: \"1ad2242c-28d7-4495-9bac-f6c8cba7f796\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc" Oct 30 23:59:24.556876 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.556865 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/63e868ff-3f36-4542-9e1d-9688ce8c37f1-device-dir\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:59:24.557265 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.556881 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1ad2242c-28d7-4495-9bac-f6c8cba7f796-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-frfhc\" (UID: \"1ad2242c-28d7-4495-9bac-f6c8cba7f796\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc" Oct 30 23:59:24.557265 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.556892 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl2s5\" (UniqueName: \"kubernetes.io/projected/63e868ff-3f36-4542-9e1d-9688ce8c37f1-kube-api-access-fl2s5\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:59:24.557265 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.556928 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfgxg\" (UniqueName: \"kubernetes.io/projected/5ab2f199-64ea-4071-b8cd-6b9861d31179-kube-api-access-qfgxg\") pod \"openshift-state-metrics-7bd7489957-tt54l\" (UID: \"5ab2f199-64ea-4071-b8cd-6b9861d31179\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tt54l" Oct 30 23:59:24.557265 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.556948 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-node-log\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.557265 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.556992 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-system-cni-dir\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.557265 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.557027 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b099593-ded8-475c-b5ef-a91bbd9aeb1a-metrics-tls\") pod \"dns-default-g5p2k\" (UID: \"8b099593-ded8-475c-b5ef-a91bbd9aeb1a\") " pod="openshift-dns/dns-default-g5p2k" Oct 30 23:59:24.557265 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.557035 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-node-exporter-accelerators-collector-config\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:24.557265 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.557037 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ad2242c-28d7-4495-9bac-f6c8cba7f796-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-frfhc\" (UID: \"1ad2242c-28d7-4495-9bac-f6c8cba7f796\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc" Oct 30 23:59:24.557265 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.557050 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d8089437-f546-441e-b121-6e79878d2608-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tc87d\" (UID: \"d8089437-f546-441e-b121-6e79878d2608\") " pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:59:24.557265 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.557076 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a0be3213-788c-45b4-8fbd-37e744a14f17-multus-daemon-config\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.557265 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.557115 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c8d0e1f0-620a-46c0-8234-628fb8acb572-telemeter-client-tls\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:24.557265 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.557157 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8tds\" (UniqueName: \"kubernetes.io/projected/1f653a44-907a-417e-a1da-9b1bb0ee7283-kube-api-access-r8tds\") pod \"iptables-alerter-zlgls\" (UID: \"1f653a44-907a-417e-a1da-9b1bb0ee7283\") " pod="openshift-network-operator/iptables-alerter-zlgls" Oct 30 23:59:24.557265 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.557208 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3cb65abb-ba0e-4075-9a0a-455e1b64b820-crio-socket\") pod \"insights-runtime-extractor-84g8j\" (UID: \"3cb65abb-ba0e-4075-9a0a-455e1b64b820\") " pod="openshift-insights/insights-runtime-extractor-84g8j" Oct 30 23:59:24.557265 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.557220 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b099593-ded8-475c-b5ef-a91bbd9aeb1a-metrics-tls\") pod \"dns-default-g5p2k\" (UID: \"8b099593-ded8-475c-b5ef-a91bbd9aeb1a\") " pod="openshift-dns/dns-default-g5p2k" Oct 30 23:59:24.557265 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.557236 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8b099593-ded8-475c-b5ef-a91bbd9aeb1a-config-volume\") pod \"dns-default-g5p2k\" (UID: \"8b099593-ded8-475c-b5ef-a91bbd9aeb1a\") " pod="openshift-dns/dns-default-g5p2k" Oct 30 23:59:24.557265 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.557246 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d8089437-f546-441e-b121-6e79878d2608-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tc87d\" (UID: \"d8089437-f546-441e-b121-6e79878d2608\") " pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:59:24.557744 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.557276 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/63e868ff-3f36-4542-9e1d-9688ce8c37f1-socket-dir\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:59:24.557744 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.557292 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a0be3213-788c-45b4-8fbd-37e744a14f17-multus-daemon-config\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.557744 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.557305 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8d0e1f0-620a-46c0-8234-628fb8acb572-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:24.557744 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.557333 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c8d0e1f0-620a-46c0-8234-628fb8acb572-telemeter-client-tls\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:24.557744 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.557343 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-systemd\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:24.557744 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.557407 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-slash\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.558095 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.558074 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8b099593-ded8-475c-b5ef-a91bbd9aeb1a-config-volume\") pod \"dns-default-g5p2k\" (UID: \"8b099593-ded8-475c-b5ef-a91bbd9aeb1a\") " pod="openshift-dns/dns-default-g5p2k" Oct 30 23:59:24.558189 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.558171 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1ad2242c-28d7-4495-9bac-f6c8cba7f796-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-frfhc\" (UID: \"1ad2242c-28d7-4495-9bac-f6c8cba7f796\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc" Oct 30 23:59:24.558280 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.558267 5274 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:59:24.558329 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.558283 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8d0e1f0-620a-46c0-8234-628fb8acb572-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:24.558377 ip-10-0-129-70 kubenswrapper[5274]: E1030 23:59:24.558332 5274 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-70.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:59:24.558419 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.558371 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/25e99515-222f-4346-9fe5-5b7b3b213ee8-ovnkube-config\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.558459 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.558436 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/25e99515-222f-4346-9fe5-5b7b3b213ee8-env-overrides\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.558557 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.558535 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1ad2242c-28d7-4495-9bac-f6c8cba7f796-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-frfhc\" (UID: \"1ad2242c-28d7-4495-9bac-f6c8cba7f796\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc" Oct 30 23:59:24.559110 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.558629 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/25e99515-222f-4346-9fe5-5b7b3b213ee8-ovn-node-metrics-cert\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.559110 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.558673 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/63e868ff-3f36-4542-9e1d-9688ce8c37f1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:59:24.559110 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.558757 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-lib-modules\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:24.559110 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.558783 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/25e99515-222f-4346-9fe5-5b7b3b213ee8-env-overrides\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.559110 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.558800 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-secret-metrics-server-client-certs\") pod \"metrics-server-6fbfb6ff44-2sxgp\" (UID: \"0a7a184e-d0ef-4c2a-ab06-129e37b7e22e\") " pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" Oct 30 23:59:24.559110 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.558833 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8d0e1f0-620a-46c0-8234-628fb8acb572-serving-certs-ca-bundle\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:24.559110 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.559058 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/25e99515-222f-4346-9fe5-5b7b3b213ee8-ovnkube-config\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.559467 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.559189 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8d0e1f0-620a-46c0-8234-628fb8acb572-serving-certs-ca-bundle\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:24.559467 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.559219 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-secret-metrics-server-client-certs\") pod \"metrics-server-6fbfb6ff44-2sxgp\" (UID: \"0a7a184e-d0ef-4c2a-ab06-129e37b7e22e\") " pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" Oct 30 23:59:24.559467 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.559250 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1f653a44-907a-417e-a1da-9b1bb0ee7283-host-slash\") pod \"iptables-alerter-zlgls\" (UID: \"1f653a44-907a-417e-a1da-9b1bb0ee7283\") " pod="openshift-network-operator/iptables-alerter-zlgls" Oct 30 23:59:24.559467 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.559300 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/99b4d92d-2003-4708-99da-a5f12cbb10e2-dbus\") pod \"global-pull-secret-syncer-6l8ht\" (UID: \"99b4d92d-2003-4708-99da-a5f12cbb10e2\") " pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:59:24.559467 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.559398 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/25e99515-222f-4346-9fe5-5b7b3b213ee8-ovn-node-metrics-cert\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.559467 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.559425 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99b4d92d-2003-4708-99da-a5f12cbb10e2-original-pull-secret\") pod \"global-pull-secret-syncer-6l8ht\" (UID: \"99b4d92d-2003-4708-99da-a5f12cbb10e2\") " pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:59:24.559715 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.559484 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/409b7296-1d35-44e9-be1a-3459f0dbaad9-agent-certs\") pod \"konnectivity-agent-x962k\" (UID: \"409b7296-1d35-44e9-be1a-3459f0dbaad9\") " pod="kube-system/konnectivity-agent-x962k" Oct 30 23:59:24.559715 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.559515 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djj7v\" (UniqueName: \"kubernetes.io/projected/0adc098d-57a4-490e-89b7-b1b271a1c087-kube-api-access-djj7v\") pod \"network-metrics-daemon-cjqfw\" (UID: \"0adc098d-57a4-490e-89b7-b1b271a1c087\") " pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:59:24.559715 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.559551 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-hostroot\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.559715 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.559674 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-client-ca-bundle\") pod \"metrics-server-6fbfb6ff44-2sxgp\" (UID: \"0a7a184e-d0ef-4c2a-ab06-129e37b7e22e\") " pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" Oct 30 23:59:24.559715 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.559708 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-modprobe-d\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:24.559971 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.559740 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-sys\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:24.559971 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.559851 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3cb65abb-ba0e-4075-9a0a-455e1b64b820-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-84g8j\" (UID: \"3cb65abb-ba0e-4075-9a0a-455e1b64b820\") " pod="openshift-insights/insights-runtime-extractor-84g8j" Oct 30 23:59:24.559971 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.559883 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8089437-f546-441e-b121-6e79878d2608-system-cni-dir\") pod \"multus-additional-cni-plugins-tc87d\" (UID: \"d8089437-f546-441e-b121-6e79878d2608\") " pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:59:24.559971 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.559914 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18698607-3254-4ec9-b3bb-bbe59c6c744f-cert\") pod \"ingress-canary-djtcv\" (UID: \"18698607-3254-4ec9-b3bb-bbe59c6c744f\") " pod="openshift-ingress-canary/ingress-canary-djtcv" Oct 30 23:59:24.559971 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.559912 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/409b7296-1d35-44e9-be1a-3459f0dbaad9-agent-certs\") pod \"konnectivity-agent-x962k\" (UID: \"409b7296-1d35-44e9-be1a-3459f0dbaad9\") " pod="kube-system/konnectivity-agent-x962k" Oct 30 23:59:24.559971 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.559945 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c8d0e1f0-620a-46c0-8234-628fb8acb572-secret-telemeter-client\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:24.560220 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.559998 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsblw\" (UniqueName: \"kubernetes.io/projected/3cb65abb-ba0e-4075-9a0a-455e1b64b820-kube-api-access-wsblw\") pod \"insights-runtime-extractor-84g8j\" (UID: \"3cb65abb-ba0e-4075-9a0a-455e1b64b820\") " pod="openshift-insights/insights-runtime-extractor-84g8j" Oct 30 23:59:24.560220 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.560116 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-audit-log\") pod \"metrics-server-6fbfb6ff44-2sxgp\" (UID: \"0a7a184e-d0ef-4c2a-ab06-129e37b7e22e\") " pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" Oct 30 23:59:24.560220 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.560143 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5405bfc2-cfc0-4ea6-80bc-911e117f1843-tmp\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:24.560220 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.560184 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-node-exporter-wtmp\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:24.560220 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.560208 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-run-openvswitch\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.560450 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.560243 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ab2f199-64ea-4071-b8cd-6b9861d31179-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-tt54l\" (UID: \"5ab2f199-64ea-4071-b8cd-6b9861d31179\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tt54l" Oct 30 23:59:24.560450 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.560279 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-host-var-lib-cni-bin\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.560450 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.560309 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/63e868ff-3f36-4542-9e1d-9688ce8c37f1-sys-fs\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:59:24.560450 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.560322 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99b4d92d-2003-4708-99da-a5f12cbb10e2-original-pull-secret\") pod \"global-pull-secret-syncer-6l8ht\" (UID: \"99b4d92d-2003-4708-99da-a5f12cbb10e2\") " pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:59:24.560450 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.560340 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-run-netns\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.561284 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.560726 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-run-ovn\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.561284 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.560764 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-etc-kubernetes\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.561284 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.560801 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znq6p\" (UniqueName: \"kubernetes.io/projected/c8d0e1f0-620a-46c0-8234-628fb8acb572-kube-api-access-znq6p\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:24.561284 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.560832 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-var-lib-kubelet\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:24.561284 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.560832 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-client-ca-bundle\") pod \"metrics-server-6fbfb6ff44-2sxgp\" (UID: \"0a7a184e-d0ef-4c2a-ab06-129e37b7e22e\") " pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" Oct 30 23:59:24.561284 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.561085 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5405bfc2-cfc0-4ea6-80bc-911e117f1843-tmp\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:24.561284 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.561101 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3cb65abb-ba0e-4075-9a0a-455e1b64b820-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-84g8j\" (UID: \"3cb65abb-ba0e-4075-9a0a-455e1b64b820\") " pod="openshift-insights/insights-runtime-extractor-84g8j" Oct 30 23:59:24.561284 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.561187 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-audit-log\") pod \"metrics-server-6fbfb6ff44-2sxgp\" (UID: \"0a7a184e-d0ef-4c2a-ab06-129e37b7e22e\") " pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" Oct 30 23:59:24.561714 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.561417 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18698607-3254-4ec9-b3bb-bbe59c6c744f-cert\") pod \"ingress-canary-djtcv\" (UID: \"18698607-3254-4ec9-b3bb-bbe59c6c744f\") " pod="openshift-ingress-canary/ingress-canary-djtcv" Oct 30 23:59:24.561714 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.561572 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c8d0e1f0-620a-46c0-8234-628fb8acb572-secret-telemeter-client\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:24.561714 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.561629 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ab2f199-64ea-4071-b8cd-6b9861d31179-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-tt54l\" (UID: \"5ab2f199-64ea-4071-b8cd-6b9861d31179\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tt54l" Oct 30 23:59:24.571061 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.571032 5274 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:59:24.571183 ip-10-0-129-70 kubenswrapper[5274]: E1030 23:59:24.571099 5274 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-70.us-east-2.compute.internal" Oct 30 23:59:24.582194 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.582158 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcvd6\" (UniqueName: \"kubernetes.io/projected/5405bfc2-cfc0-4ea6-80bc-911e117f1843-kube-api-access-mcvd6\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:24.582450 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.582430 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8wrl\" (UniqueName: \"kubernetes.io/projected/d8089437-f546-441e-b121-6e79878d2608-kube-api-access-d8wrl\") pod \"multus-additional-cni-plugins-tc87d\" (UID: \"d8089437-f546-441e-b121-6e79878d2608\") " pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:59:24.583376 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.583340 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85v2m\" (UniqueName: \"kubernetes.io/projected/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-kube-api-access-85v2m\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:24.583376 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.583367 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpxpg\" (UniqueName: \"kubernetes.io/projected/25e99515-222f-4346-9fe5-5b7b3b213ee8-kube-api-access-dpxpg\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.583536 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.583505 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znq6p\" (UniqueName: \"kubernetes.io/projected/c8d0e1f0-620a-46c0-8234-628fb8acb572-kube-api-access-znq6p\") pod \"telemeter-client-5dc5757857-pp2xd\" (UID: \"c8d0e1f0-620a-46c0-8234-628fb8acb572\") " pod="openshift-monitoring/telemeter-client-5dc5757857-pp2xd" Oct 30 23:59:24.583646 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.583626 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gprnc\" (UniqueName: \"kubernetes.io/projected/1ad2242c-28d7-4495-9bac-f6c8cba7f796-kube-api-access-gprnc\") pod \"kube-state-metrics-5dc8d7d89b-frfhc\" (UID: \"1ad2242c-28d7-4495-9bac-f6c8cba7f796\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-frfhc" Oct 30 23:59:24.583711 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.583695 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd4j2\" (UniqueName: \"kubernetes.io/projected/50c15c32-e5cd-426c-9e7d-c123f8480653-kube-api-access-gd4j2\") pod \"network-check-target-bd4bx\" (UID: \"50c15c32-e5cd-426c-9e7d-c123f8480653\") " pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:59:24.584527 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.584510 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnlg7\" (UniqueName: \"kubernetes.io/projected/8b099593-ded8-475c-b5ef-a91bbd9aeb1a-kube-api-access-gnlg7\") pod \"dns-default-g5p2k\" (UID: \"8b099593-ded8-475c-b5ef-a91bbd9aeb1a\") " pod="openshift-dns/dns-default-g5p2k" Oct 30 23:59:24.586720 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.586700 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djj7v\" (UniqueName: \"kubernetes.io/projected/0adc098d-57a4-490e-89b7-b1b271a1c087-kube-api-access-djj7v\") pod \"network-metrics-daemon-cjqfw\" (UID: \"0adc098d-57a4-490e-89b7-b1b271a1c087\") " pod="openshift-multus/network-metrics-daemon-cjqfw" Oct 30 23:59:24.587071 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.587056 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqkgh\" (UniqueName: \"kubernetes.io/projected/18698607-3254-4ec9-b3bb-bbe59c6c744f-kube-api-access-cqkgh\") pod \"ingress-canary-djtcv\" (UID: \"18698607-3254-4ec9-b3bb-bbe59c6c744f\") " pod="openshift-ingress-canary/ingress-canary-djtcv" Oct 30 23:59:24.587372 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.587346 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kjxh\" (UniqueName: \"kubernetes.io/projected/d7815306-7a99-4fc5-8226-b34916ceaaa2-kube-api-access-2kjxh\") pod \"node-resolver-6hq7b\" (UID: \"d7815306-7a99-4fc5-8226-b34916ceaaa2\") " pod="openshift-dns/node-resolver-6hq7b" Oct 30 23:59:24.587626 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.587607 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlqqp\" (UniqueName: \"kubernetes.io/projected/a0be3213-788c-45b4-8fbd-37e744a14f17-kube-api-access-hlqqp\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.587766 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.587748 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fr52\" (UniqueName: \"kubernetes.io/projected/0a7a184e-d0ef-4c2a-ab06-129e37b7e22e-kube-api-access-5fr52\") pod \"metrics-server-6fbfb6ff44-2sxgp\" (UID: \"0a7a184e-d0ef-4c2a-ab06-129e37b7e22e\") " pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" Oct 30 23:59:24.587982 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.587942 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmcgf\" (UniqueName: \"kubernetes.io/projected/18acd0a8-988e-4168-84e7-0b25dc7b32d4-kube-api-access-pmcgf\") pod \"node-ca-vnk6j\" (UID: \"18acd0a8-988e-4168-84e7-0b25dc7b32d4\") " pod="openshift-image-registry/node-ca-vnk6j" Oct 30 23:59:24.588370 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.588350 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl2s5\" (UniqueName: \"kubernetes.io/projected/63e868ff-3f36-4542-9e1d-9688ce8c37f1-kube-api-access-fl2s5\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:59:24.588605 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.588583 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8tds\" (UniqueName: \"kubernetes.io/projected/1f653a44-907a-417e-a1da-9b1bb0ee7283-kube-api-access-r8tds\") pod \"iptables-alerter-zlgls\" (UID: \"1f653a44-907a-417e-a1da-9b1bb0ee7283\") " pod="openshift-network-operator/iptables-alerter-zlgls" Oct 30 23:59:24.588658 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.588645 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsblw\" (UniqueName: \"kubernetes.io/projected/3cb65abb-ba0e-4075-9a0a-455e1b64b820-kube-api-access-wsblw\") pod \"insights-runtime-extractor-84g8j\" (UID: \"3cb65abb-ba0e-4075-9a0a-455e1b64b820\") " pod="openshift-insights/insights-runtime-extractor-84g8j" Oct 30 23:59:24.591089 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.591070 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfgxg\" (UniqueName: \"kubernetes.io/projected/5ab2f199-64ea-4071-b8cd-6b9861d31179-kube-api-access-qfgxg\") pod \"openshift-state-metrics-7bd7489957-tt54l\" (UID: \"5ab2f199-64ea-4071-b8cd-6b9861d31179\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tt54l" Oct 30 23:59:24.661447 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.661408 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/99b4d92d-2003-4708-99da-a5f12cbb10e2-dbus\") pod \"global-pull-secret-syncer-6l8ht\" (UID: \"99b4d92d-2003-4708-99da-a5f12cbb10e2\") " pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:59:24.661447 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.661454 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-hostroot\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.661679 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.661481 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-modprobe-d\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:24.661679 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.661503 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-sys\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:24.661679 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.661547 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/99b4d92d-2003-4708-99da-a5f12cbb10e2-dbus\") pod \"global-pull-secret-syncer-6l8ht\" (UID: \"99b4d92d-2003-4708-99da-a5f12cbb10e2\") " pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:59:24.661679 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.661550 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-hostroot\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.661679 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.661558 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8089437-f546-441e-b121-6e79878d2608-system-cni-dir\") pod \"multus-additional-cni-plugins-tc87d\" (UID: \"d8089437-f546-441e-b121-6e79878d2608\") " pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:59:24.661679 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.661594 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8089437-f546-441e-b121-6e79878d2608-system-cni-dir\") pod \"multus-additional-cni-plugins-tc87d\" (UID: \"d8089437-f546-441e-b121-6e79878d2608\") " pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:59:24.661679 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.661602 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-sys\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:24.661679 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.661626 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-node-exporter-wtmp\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:24.661679 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.661641 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-modprobe-d\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:24.661679 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.661653 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-run-openvswitch\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.661679 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.661676 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-host-var-lib-cni-bin\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.661689 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-node-exporter-wtmp\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.661701 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/63e868ff-3f36-4542-9e1d-9688ce8c37f1-sys-fs\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.661705 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-run-openvswitch\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.661727 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-run-netns\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.661735 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-host-var-lib-cni-bin\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.661750 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-run-ovn\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.661758 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/63e868ff-3f36-4542-9e1d-9688ce8c37f1-sys-fs\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.661773 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-etc-kubernetes\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.661778 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-run-netns\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.661795 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-run-ovn\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.661808 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-etc-kubernetes\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.661801 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-var-lib-kubelet\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.661843 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-run-systemd\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.661863 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-host-var-lib-cni-multus\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.661886 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-var-lib-kubelet\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.661901 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-run-systemd\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.661909 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-host-var-lib-cni-multus\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.661922 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-host\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.661949 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-cnibin\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.661992 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-systemd-units\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662005 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-host\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662018 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-cni-netd\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662041 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-multus-cni-dir\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662041 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-systemd-units\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662064 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-sysconfig\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662060 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-cnibin\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662066 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-cni-netd\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662085 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-run\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662103 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-multus-cni-dir\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662119 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-host-run-multus-certs\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662142 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-sysconfig\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662148 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8089437-f546-441e-b121-6e79878d2608-cnibin\") pod \"multus-additional-cni-plugins-tc87d\" (UID: \"d8089437-f546-441e-b121-6e79878d2608\") " pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662157 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-run\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662173 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-host-var-lib-kubelet\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662193 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-host-run-multus-certs\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662196 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8089437-f546-441e-b121-6e79878d2608-cnibin\") pod \"multus-additional-cni-plugins-tc87d\" (UID: \"d8089437-f546-441e-b121-6e79878d2608\") " pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662196 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/18acd0a8-988e-4168-84e7-0b25dc7b32d4-host\") pod \"node-ca-vnk6j\" (UID: \"18acd0a8-988e-4168-84e7-0b25dc7b32d4\") " pod="openshift-image-registry/node-ca-vnk6j" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662226 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/18acd0a8-988e-4168-84e7-0b25dc7b32d4-host\") pod \"node-ca-vnk6j\" (UID: \"18acd0a8-988e-4168-84e7-0b25dc7b32d4\") " pod="openshift-image-registry/node-ca-vnk6j" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662226 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-host-var-lib-kubelet\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.662225 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662254 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662292 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662301 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-sysctl-conf\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662324 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-etc-openvswitch\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662345 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-host-run-k8s-cni-cncf-io\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662366 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-kubelet\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662379 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-etc-openvswitch\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662385 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-multus-conf-dir\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662414 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-multus-conf-dir\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662426 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-sysctl-conf\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662437 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-cni-bin\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662456 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-kubelet\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662461 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-host-run-netns\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662464 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-host-run-k8s-cni-cncf-io\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662485 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-cni-bin\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662486 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8089437-f546-441e-b121-6e79878d2608-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tc87d\" (UID: \"d8089437-f546-441e-b121-6e79878d2608\") " pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662529 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-root\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662576 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8089437-f546-441e-b121-6e79878d2608-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tc87d\" (UID: \"d8089437-f546-441e-b121-6e79878d2608\") " pod="openshift-multus/multus-additional-cni-plugins-tc87d" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662579 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-log-socket\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662604 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-host-run-netns\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662632 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-run-ovn-kubernetes\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662654 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/63e868ff-3f36-4542-9e1d-9688ce8c37f1-etc-selinux\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662698 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-run-ovn-kubernetes\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662706 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-root\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662720 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-var-lib-openvswitch\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662729 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-log-socket\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662739 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/63e868ff-3f36-4542-9e1d-9688ce8c37f1-etc-selinux\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662774 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-multus-socket-dir-parent\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662806 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-sysctl-d\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662818 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-multus-socket-dir-parent\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662841 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-sys\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662860 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-var-lib-openvswitch\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662876 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/63e868ff-3f36-4542-9e1d-9688ce8c37f1-registration-dir\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662922 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-kubernetes\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662932 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/63e868ff-3f36-4542-9e1d-9688ce8c37f1-registration-dir\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662935 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-sysctl-d\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662971 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d7815306-7a99-4fc5-8226-b34916ceaaa2-hosts-file\") pod \"node-resolver-6hq7b\" (UID: \"d7815306-7a99-4fc5-8226-b34916ceaaa2\") " pod="openshift-dns/node-resolver-6hq7b" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662997 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/99b4d92d-2003-4708-99da-a5f12cbb10e2-kubelet-config\") pod \"global-pull-secret-syncer-6l8ht\" (UID: \"99b4d92d-2003-4708-99da-a5f12cbb10e2\") " pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.663013 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d7815306-7a99-4fc5-8226-b34916ceaaa2-hosts-file\") pod \"node-resolver-6hq7b\" (UID: \"d7815306-7a99-4fc5-8226-b34916ceaaa2\") " pod="openshift-dns/node-resolver-6hq7b" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.662973 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9-sys\") pod \"node-exporter-bpczc\" (UID: \"de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9\") " pod="openshift-monitoring/node-exporter-bpczc" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.663030 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/63e868ff-3f36-4542-9e1d-9688ce8c37f1-device-dir\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.663055 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-node-log\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.663072 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-system-cni-dir\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.663087 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/99b4d92d-2003-4708-99da-a5f12cbb10e2-kubelet-config\") pod \"global-pull-secret-syncer-6l8ht\" (UID: \"99b4d92d-2003-4708-99da-a5f12cbb10e2\") " pod="kube-system/global-pull-secret-syncer-6l8ht" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.663091 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3cb65abb-ba0e-4075-9a0a-455e1b64b820-crio-socket\") pod \"insights-runtime-extractor-84g8j\" (UID: \"3cb65abb-ba0e-4075-9a0a-455e1b64b820\") " pod="openshift-insights/insights-runtime-extractor-84g8j" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.663116 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/63e868ff-3f36-4542-9e1d-9688ce8c37f1-device-dir\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.663002 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-kubernetes\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.663137 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-node-log\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.663142 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0be3213-788c-45b4-8fbd-37e744a14f17-system-cni-dir\") pod \"multus-9j82q\" (UID: \"a0be3213-788c-45b4-8fbd-37e744a14f17\") " pod="openshift-multus/multus-9j82q" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.663157 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/63e868ff-3f36-4542-9e1d-9688ce8c37f1-socket-dir\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.663192 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3cb65abb-ba0e-4075-9a0a-455e1b64b820-crio-socket\") pod \"insights-runtime-extractor-84g8j\" (UID: \"3cb65abb-ba0e-4075-9a0a-455e1b64b820\") " pod="openshift-insights/insights-runtime-extractor-84g8j" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.663223 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-systemd\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.663247 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/63e868ff-3f36-4542-9e1d-9688ce8c37f1-socket-dir\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.663284 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-slash\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.663304 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25e99515-222f-4346-9fe5-5b7b3b213ee8-host-slash\") pod \"ovnkube-node-7zxsz\" (UID: \"25e99515-222f-4346-9fe5-5b7b3b213ee8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.663354 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-etc-systemd\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.663389 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/63e868ff-3f36-4542-9e1d-9688ce8c37f1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.663415 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-lib-modules\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.663438 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1f653a44-907a-417e-a1da-9b1bb0ee7283-host-slash\") pod \"iptables-alerter-zlgls\" (UID: \"1f653a44-907a-417e-a1da-9b1bb0ee7283\") " pod="openshift-network-operator/iptables-alerter-zlgls" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.663485 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/63e868ff-3f36-4542-9e1d-9688ce8c37f1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gn89w\" (UID: \"63e868ff-3f36-4542-9e1d-9688ce8c37f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gn89w" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.663499 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1f653a44-907a-417e-a1da-9b1bb0ee7283-host-slash\") pod \"iptables-alerter-zlgls\" (UID: \"1f653a44-907a-417e-a1da-9b1bb0ee7283\") " pod="openshift-network-operator/iptables-alerter-zlgls" Oct 30 23:59:24.664111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:24.663559 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5405bfc2-cfc0-4ea6-80bc-911e117f1843-lib-modules\") pod \"tuned-f64c8\" (UID: \"5405bfc2-cfc0-4ea6-80bc-911e117f1843\") " pod="openshift-cluster-node-tuning-operator/tuned-f64c8" Oct 30 23:59:26.371538 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:26.371503 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 30 23:59:26.499472 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:26.499444 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:27.056168 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:27.056140 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-g5p2k" Oct 30 23:59:31.569915 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:31.569883 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:31.570382 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:31.570121 5274 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 23:59:39.500693 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:39.500616 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6fbfb6ff44-2sxgp" Oct 30 23:59:46.350833 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.350796 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 23:59:46.356514 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.356490 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.364166 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.364144 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 23:59:46.364261 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.364189 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 23:59:46.364372 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.364354 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 23:59:46.364413 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.364386 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 23:59:46.365504 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.365488 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-f8xwm\"" Oct 30 23:59:46.365562 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.365522 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 23:59:46.365562 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.365527 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 23:59:46.365562 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.365543 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 23:59:46.365652 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.365646 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 23:59:46.369649 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.369627 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 23:59:46.371610 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.371590 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 23:59:46.400293 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.400270 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1ecb2f10-eefd-4680-8741-3f16a479fb2a-config-volume\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.400393 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.400300 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1ecb2f10-eefd-4680-8741-3f16a479fb2a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.400393 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.400319 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ecb2f10-eefd-4680-8741-3f16a479fb2a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.400393 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.400335 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1ecb2f10-eefd-4680-8741-3f16a479fb2a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.400536 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.400391 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1ecb2f10-eefd-4680-8741-3f16a479fb2a-config-out\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.400536 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.400432 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1ecb2f10-eefd-4680-8741-3f16a479fb2a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.400536 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.400526 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1ecb2f10-eefd-4680-8741-3f16a479fb2a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.400626 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.400561 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmk6h\" (UniqueName: \"kubernetes.io/projected/1ecb2f10-eefd-4680-8741-3f16a479fb2a-kube-api-access-pmk6h\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.400626 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.400589 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ecb2f10-eefd-4680-8741-3f16a479fb2a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.400688 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.400644 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1ecb2f10-eefd-4680-8741-3f16a479fb2a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.400688 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.400674 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1ecb2f10-eefd-4680-8741-3f16a479fb2a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.400745 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.400700 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1ecb2f10-eefd-4680-8741-3f16a479fb2a-web-config\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.400787 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.400774 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1ecb2f10-eefd-4680-8741-3f16a479fb2a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.501290 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.501263 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1ecb2f10-eefd-4680-8741-3f16a479fb2a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.501412 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.501300 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmk6h\" (UniqueName: \"kubernetes.io/projected/1ecb2f10-eefd-4680-8741-3f16a479fb2a-kube-api-access-pmk6h\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.501412 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.501332 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ecb2f10-eefd-4680-8741-3f16a479fb2a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.501412 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.501359 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1ecb2f10-eefd-4680-8741-3f16a479fb2a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.501412 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.501396 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1ecb2f10-eefd-4680-8741-3f16a479fb2a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.501610 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.501426 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1ecb2f10-eefd-4680-8741-3f16a479fb2a-web-config\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.501610 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.501452 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1ecb2f10-eefd-4680-8741-3f16a479fb2a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.501610 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.501479 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1ecb2f10-eefd-4680-8741-3f16a479fb2a-config-volume\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.501610 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.501508 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1ecb2f10-eefd-4680-8741-3f16a479fb2a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.501775 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.501696 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1ecb2f10-eefd-4680-8741-3f16a479fb2a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.501775 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.501728 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ecb2f10-eefd-4680-8741-3f16a479fb2a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.501775 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.501747 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1ecb2f10-eefd-4680-8741-3f16a479fb2a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.501922 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.501778 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1ecb2f10-eefd-4680-8741-3f16a479fb2a-config-out\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.501922 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.501798 5274 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 23:59:46.502586 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.502559 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ecb2f10-eefd-4680-8741-3f16a479fb2a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.502744 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.501807 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1ecb2f10-eefd-4680-8741-3f16a479fb2a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.503021 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.502990 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ecb2f10-eefd-4680-8741-3f16a479fb2a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.504498 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.504453 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1ecb2f10-eefd-4680-8741-3f16a479fb2a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.505099 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.505000 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1ecb2f10-eefd-4680-8741-3f16a479fb2a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.505099 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.505029 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1ecb2f10-eefd-4680-8741-3f16a479fb2a-config-out\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.505099 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.505066 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1ecb2f10-eefd-4680-8741-3f16a479fb2a-web-config\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.505314 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.505133 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1ecb2f10-eefd-4680-8741-3f16a479fb2a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.505314 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.505131 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1ecb2f10-eefd-4680-8741-3f16a479fb2a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.505314 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.505265 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1ecb2f10-eefd-4680-8741-3f16a479fb2a-config-volume\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.505872 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.505855 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1ecb2f10-eefd-4680-8741-3f16a479fb2a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.506379 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.506359 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1ecb2f10-eefd-4680-8741-3f16a479fb2a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.513595 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.513578 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmk6h\" (UniqueName: \"kubernetes.io/projected/1ecb2f10-eefd-4680-8741-3f16a479fb2a-kube-api-access-pmk6h\") pod \"alertmanager-main-0\" (UID: \"1ecb2f10-eefd-4680-8741-3f16a479fb2a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.683898 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.683865 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.972111 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:46.972083 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 23:59:46.974769 ip-10-0-129-70 kubenswrapper[5274]: W1030 23:59:46.974741 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ecb2f10_eefd_4680_8741_3f16a479fb2a.slice/crio-e6f7375ae8e7a9671105ad9e7264456bbe9ff80d1aec1c5e3970d762dae2d373 WatchSource:0}: Error finding container e6f7375ae8e7a9671105ad9e7264456bbe9ff80d1aec1c5e3970d762dae2d373: Status 404 returned error can't find the container with id e6f7375ae8e7a9671105ad9e7264456bbe9ff80d1aec1c5e3970d762dae2d373 Oct 30 23:59:47.628112 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:47.628071 5274 generic.go:356] "Generic (PLEG): container finished" podID="1ecb2f10-eefd-4680-8741-3f16a479fb2a" containerID="b1e9c9b6dc832bd815bce47629126426d150639ef52f9d8849e51a85f445c264" exitCode=0 Oct 30 23:59:47.628631 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:47.628121 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ecb2f10-eefd-4680-8741-3f16a479fb2a","Type":"ContainerDied","Data":"b1e9c9b6dc832bd815bce47629126426d150639ef52f9d8849e51a85f445c264"} Oct 30 23:59:47.628631 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:47.628175 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ecb2f10-eefd-4680-8741-3f16a479fb2a","Type":"ContainerStarted","Data":"e6f7375ae8e7a9671105ad9e7264456bbe9ff80d1aec1c5e3970d762dae2d373"} Oct 30 23:59:47.629296 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:47.629274 5274 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 23:59:49.638447 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:49.638411 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ecb2f10-eefd-4680-8741-3f16a479fb2a","Type":"ContainerStarted","Data":"0dcff05c5ad25577ae04aa9656d844114bc605b51f881d377ca0273016e2074e"} Oct 30 23:59:49.638827 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:49.638481 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ecb2f10-eefd-4680-8741-3f16a479fb2a","Type":"ContainerStarted","Data":"0250d14ffb87e4c0d576845d973fec6a8352567ec53136f3be3298772d0256ad"} Oct 30 23:59:49.638827 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:49.638506 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ecb2f10-eefd-4680-8741-3f16a479fb2a","Type":"ContainerStarted","Data":"eaadab97f7c1ee3217638669d4d7bc2d0a80c36112a340dd5307120379f40fb3"} Oct 30 23:59:49.638827 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:49.638529 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ecb2f10-eefd-4680-8741-3f16a479fb2a","Type":"ContainerStarted","Data":"edb7fb469c0fd3ce9d233ab55d2dce00a3d6b747ab65fc2d80351822a7034568"} Oct 30 23:59:49.638827 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:49.638552 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ecb2f10-eefd-4680-8741-3f16a479fb2a","Type":"ContainerStarted","Data":"f1d400deb1eba42675863b67dcf3435f541357fedf248d74da8e1c2919b48abe"} Oct 30 23:59:50.647481 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:50.647449 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1ecb2f10-eefd-4680-8741-3f16a479fb2a","Type":"ContainerStarted","Data":"4f1387bf2494dbcfd608a7a145c564837b218bf62537ce81e36be357cc4935ca"} Oct 30 23:59:50.686983 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:50.686918 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.651474402 podStartE2EDuration="4.686905675s" totalImagesPullingTime="2.02386002s" totalInitContainerRuntime="11.571253ms" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:50.685300774 +0000 UTC m=+27.322119489" watchObservedRunningTime="2026-10-30 23:59:50.686905675 +0000 UTC m=+27.323724388" Oct 30 23:59:50.708305 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:50.708281 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-x962k" Oct 30 23:59:50.899672 ip-10-0-129-70 kubenswrapper[5274]: E1030 23:59:50.899597 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 30 23:59:51.964009 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:51.963972 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7zxsz" Oct 30 23:59:56.374546 ip-10-0-129-70 kubenswrapper[5274]: I1030 23:59:56.374508 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bd4bx" Oct 31 00:00:00.173126 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:00:00.169978 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-pruner-29890080-6f25s"] Oct 31 00:00:00.175485 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:00:00.175460 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29890080-6f25s" Oct 31 00:00:00.184930 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:00:00.184897 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29890080-6f25s"] Oct 31 00:00:00.185520 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:00:00.185498 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"serviceca\"" Oct 31 00:00:00.185520 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:00:00.185518 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"pruner-dockercfg-5j78z\"" Oct 31 00:00:00.310947 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:00:00.310909 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-527gp\" (UniqueName: \"kubernetes.io/projected/3401aa52-8809-45ca-9e99-02fc01a6639d-kube-api-access-527gp\") pod \"image-pruner-29890080-6f25s\" (UID: \"3401aa52-8809-45ca-9e99-02fc01a6639d\") " pod="openshift-image-registry/image-pruner-29890080-6f25s" Oct 31 00:00:00.311117 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:00:00.310971 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3401aa52-8809-45ca-9e99-02fc01a6639d-serviceca\") pod \"image-pruner-29890080-6f25s\" (UID: \"3401aa52-8809-45ca-9e99-02fc01a6639d\") " pod="openshift-image-registry/image-pruner-29890080-6f25s" Oct 31 00:00:00.411431 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:00:00.411396 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3401aa52-8809-45ca-9e99-02fc01a6639d-serviceca\") pod \"image-pruner-29890080-6f25s\" (UID: \"3401aa52-8809-45ca-9e99-02fc01a6639d\") " pod="openshift-image-registry/image-pruner-29890080-6f25s" Oct 31 00:00:00.411630 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:00:00.411464 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-527gp\" (UniqueName: \"kubernetes.io/projected/3401aa52-8809-45ca-9e99-02fc01a6639d-kube-api-access-527gp\") pod \"image-pruner-29890080-6f25s\" (UID: \"3401aa52-8809-45ca-9e99-02fc01a6639d\") " pod="openshift-image-registry/image-pruner-29890080-6f25s" Oct 31 00:00:00.412099 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:00:00.412075 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3401aa52-8809-45ca-9e99-02fc01a6639d-serviceca\") pod \"image-pruner-29890080-6f25s\" (UID: \"3401aa52-8809-45ca-9e99-02fc01a6639d\") " pod="openshift-image-registry/image-pruner-29890080-6f25s" Oct 31 00:00:00.425556 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:00:00.425494 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-527gp\" (UniqueName: \"kubernetes.io/projected/3401aa52-8809-45ca-9e99-02fc01a6639d-kube-api-access-527gp\") pod \"image-pruner-29890080-6f25s\" (UID: \"3401aa52-8809-45ca-9e99-02fc01a6639d\") " pod="openshift-image-registry/image-pruner-29890080-6f25s" Oct 31 00:00:00.515060 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:00:00.515022 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29890080-6f25s" Oct 31 00:00:00.785227 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:00:00.785149 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29890080-6f25s"] Oct 31 00:00:00.788810 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:00:00.788776 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3401aa52_8809_45ca_9e99_02fc01a6639d.slice/crio-4f0cc9e54f6f50ae0fbb944d3977bde1627d1bff1986ad4381146efb7546fb0d WatchSource:0}: Error finding container 4f0cc9e54f6f50ae0fbb944d3977bde1627d1bff1986ad4381146efb7546fb0d: Status 404 returned error can't find the container with id 4f0cc9e54f6f50ae0fbb944d3977bde1627d1bff1986ad4381146efb7546fb0d Oct 31 00:00:01.683136 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:00:01.683057 5274 generic.go:356] "Generic (PLEG): container finished" podID="3401aa52-8809-45ca-9e99-02fc01a6639d" containerID="9fa3849f1bd904b383a3db08a35e5a5f7ae962c7469614315da5a0445ec0a8d5" exitCode=0 Oct 31 00:00:01.683136 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:00:01.683096 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29890080-6f25s" event={"ID":"3401aa52-8809-45ca-9e99-02fc01a6639d","Type":"ContainerDied","Data":"9fa3849f1bd904b383a3db08a35e5a5f7ae962c7469614315da5a0445ec0a8d5"} Oct 31 00:00:01.683136 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:00:01.683131 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29890080-6f25s" event={"ID":"3401aa52-8809-45ca-9e99-02fc01a6639d","Type":"ContainerStarted","Data":"4f0cc9e54f6f50ae0fbb944d3977bde1627d1bff1986ad4381146efb7546fb0d"} Oct 31 00:00:02.815556 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:00:02.815529 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29890080-6f25s" Oct 31 00:00:02.930105 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:00:02.930076 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3401aa52-8809-45ca-9e99-02fc01a6639d-kube-api-access-527gp\" (UniqueName: \"kubernetes.io/projected/3401aa52-8809-45ca-9e99-02fc01a6639d-kube-api-access-527gp\") pod \"3401aa52-8809-45ca-9e99-02fc01a6639d\" (UID: \"3401aa52-8809-45ca-9e99-02fc01a6639d\") " Oct 31 00:00:02.930256 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:00:02.930178 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3401aa52-8809-45ca-9e99-02fc01a6639d-serviceca\" (UniqueName: \"kubernetes.io/configmap/3401aa52-8809-45ca-9e99-02fc01a6639d-serviceca\") pod \"3401aa52-8809-45ca-9e99-02fc01a6639d\" (UID: \"3401aa52-8809-45ca-9e99-02fc01a6639d\") " Oct 31 00:00:02.930668 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:00:02.930645 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3401aa52-8809-45ca-9e99-02fc01a6639d-serviceca" pod "3401aa52-8809-45ca-9e99-02fc01a6639d" (UID: "3401aa52-8809-45ca-9e99-02fc01a6639d"). InnerVolumeSpecName "serviceca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:00:02.932329 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:00:02.932309 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3401aa52-8809-45ca-9e99-02fc01a6639d-kube-api-access-527gp" pod "3401aa52-8809-45ca-9e99-02fc01a6639d" (UID: "3401aa52-8809-45ca-9e99-02fc01a6639d"). InnerVolumeSpecName "kube-api-access-527gp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:00:03.031024 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:00:03.030917 5274 reconciler_common.go:299] "Volume detached for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3401aa52-8809-45ca-9e99-02fc01a6639d-serviceca\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:03.031024 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:00:03.030950 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-527gp\" (UniqueName: \"kubernetes.io/projected/3401aa52-8809-45ca-9e99-02fc01a6639d-kube-api-access-527gp\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:00:03.690301 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:00:03.690268 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29890080-6f25s" event={"ID":"3401aa52-8809-45ca-9e99-02fc01a6639d","Type":"ContainerDied","Data":"4f0cc9e54f6f50ae0fbb944d3977bde1627d1bff1986ad4381146efb7546fb0d"} Oct 31 00:00:03.690301 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:00:03.690291 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29890080-6f25s" Oct 31 00:00:03.690519 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:00:03.690312 5274 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f0cc9e54f6f50ae0fbb944d3977bde1627d1bff1986ad4381146efb7546fb0d" Oct 31 00:00:20.898783 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:00:20.898743 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:00:50.898849 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:00:50.898809 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:01:20.898780 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:01:20.898701 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:01:42.390526 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:01:42.390494 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpx4p"] Oct 31 00:01:42.391021 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:01:42.390792 5274 cpu_manager.go:414] "containerMap: removing container" podUID="3401aa52-8809-45ca-9e99-02fc01a6639d" containerName="image-pruner" Oct 31 00:01:42.391021 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:01:42.390803 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3401aa52-8809-45ca-9e99-02fc01a6639d" containerName="image-pruner" Oct 31 00:01:42.391021 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:01:42.390854 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="3401aa52-8809-45ca-9e99-02fc01a6639d" containerName="image-pruner" Oct 31 00:01:42.393774 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:01:42.393758 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpx4p" Oct 31 00:01:42.398681 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:01:42.398659 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 31 00:01:42.398784 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:01:42.398691 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-5mwlt\"" Oct 31 00:01:42.398841 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:01:42.398811 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 31 00:01:42.406344 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:01:42.406327 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpx4p"] Oct 31 00:01:42.511754 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:01:42.511722 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e28530d-6eb5-42ae-885d-5100c7d76ae2-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpx4p\" (UID: \"8e28530d-6eb5-42ae-885d-5100c7d76ae2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpx4p" Oct 31 00:01:42.511895 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:01:42.511758 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrb79\" (UniqueName: \"kubernetes.io/projected/8e28530d-6eb5-42ae-885d-5100c7d76ae2-kube-api-access-zrb79\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpx4p\" (UID: \"8e28530d-6eb5-42ae-885d-5100c7d76ae2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpx4p" Oct 31 00:01:42.511895 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:01:42.511872 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e28530d-6eb5-42ae-885d-5100c7d76ae2-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpx4p\" (UID: \"8e28530d-6eb5-42ae-885d-5100c7d76ae2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpx4p" Oct 31 00:01:42.612525 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:01:42.612496 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e28530d-6eb5-42ae-885d-5100c7d76ae2-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpx4p\" (UID: \"8e28530d-6eb5-42ae-885d-5100c7d76ae2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpx4p" Oct 31 00:01:42.612647 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:01:42.612528 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrb79\" (UniqueName: \"kubernetes.io/projected/8e28530d-6eb5-42ae-885d-5100c7d76ae2-kube-api-access-zrb79\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpx4p\" (UID: \"8e28530d-6eb5-42ae-885d-5100c7d76ae2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpx4p" Oct 31 00:01:42.612647 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:01:42.612562 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e28530d-6eb5-42ae-885d-5100c7d76ae2-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpx4p\" (UID: \"8e28530d-6eb5-42ae-885d-5100c7d76ae2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpx4p" Oct 31 00:01:42.612876 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:01:42.612859 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e28530d-6eb5-42ae-885d-5100c7d76ae2-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpx4p\" (UID: \"8e28530d-6eb5-42ae-885d-5100c7d76ae2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpx4p" Oct 31 00:01:42.612950 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:01:42.612913 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e28530d-6eb5-42ae-885d-5100c7d76ae2-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpx4p\" (UID: \"8e28530d-6eb5-42ae-885d-5100c7d76ae2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpx4p" Oct 31 00:01:42.625299 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:01:42.625280 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrb79\" (UniqueName: \"kubernetes.io/projected/8e28530d-6eb5-42ae-885d-5100c7d76ae2-kube-api-access-zrb79\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpx4p\" (UID: \"8e28530d-6eb5-42ae-885d-5100c7d76ae2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpx4p" Oct 31 00:01:42.709978 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:01:42.709904 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpx4p" Oct 31 00:01:42.981017 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:01:42.980896 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpx4p"] Oct 31 00:01:42.983914 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:01:42.983871 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e28530d_6eb5_42ae_885d_5100c7d76ae2.slice/crio-3c0162efff32aa6f063641131d6028285e2f84d7c283e20cf8bde48c94d143ae WatchSource:0}: Error finding container 3c0162efff32aa6f063641131d6028285e2f84d7c283e20cf8bde48c94d143ae: Status 404 returned error can't find the container with id 3c0162efff32aa6f063641131d6028285e2f84d7c283e20cf8bde48c94d143ae Oct 31 00:01:43.018214 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:01:43.018192 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpx4p" event={"ID":"8e28530d-6eb5-42ae-885d-5100c7d76ae2","Type":"ContainerStarted","Data":"3c0162efff32aa6f063641131d6028285e2f84d7c283e20cf8bde48c94d143ae"} Oct 31 00:01:49.042737 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:01:49.042701 5274 generic.go:356] "Generic (PLEG): container finished" podID="8e28530d-6eb5-42ae-885d-5100c7d76ae2" containerID="8667830aaa1dcc1aa76901b31d58f3c233a728685b28a1bcfc013dde49b1e311" exitCode=0 Oct 31 00:01:49.043161 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:01:49.042752 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpx4p" event={"ID":"8e28530d-6eb5-42ae-885d-5100c7d76ae2","Type":"ContainerDied","Data":"8667830aaa1dcc1aa76901b31d58f3c233a728685b28a1bcfc013dde49b1e311"} Oct 31 00:01:50.904139 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:01:50.904101 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:01:52.057541 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:01:52.057502 5274 generic.go:356] "Generic (PLEG): container finished" podID="8e28530d-6eb5-42ae-885d-5100c7d76ae2" containerID="ee86714b778250b0a2e05833e21223df4a26b66b42af1cd7ec0d30762db88a9c" exitCode=0 Oct 31 00:01:52.057903 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:01:52.057568 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpx4p" event={"ID":"8e28530d-6eb5-42ae-885d-5100c7d76ae2","Type":"ContainerDied","Data":"ee86714b778250b0a2e05833e21223df4a26b66b42af1cd7ec0d30762db88a9c"} Oct 31 00:01:59.085048 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:01:59.084965 5274 generic.go:356] "Generic (PLEG): container finished" podID="8e28530d-6eb5-42ae-885d-5100c7d76ae2" containerID="cfe989a74d27d8dff2aa47cb54d97058d23dddf1ac367bb2e42ea22f7a6870fa" exitCode=0 Oct 31 00:01:59.085048 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:01:59.085011 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpx4p" event={"ID":"8e28530d-6eb5-42ae-885d-5100c7d76ae2","Type":"ContainerDied","Data":"cfe989a74d27d8dff2aa47cb54d97058d23dddf1ac367bb2e42ea22f7a6870fa"} Oct 31 00:02:00.202788 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:00.202763 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpx4p" Oct 31 00:02:00.350940 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:00.350880 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e28530d-6eb5-42ae-885d-5100c7d76ae2-kube-api-access-zrb79\" (UniqueName: \"kubernetes.io/projected/8e28530d-6eb5-42ae-885d-5100c7d76ae2-kube-api-access-zrb79\") pod \"8e28530d-6eb5-42ae-885d-5100c7d76ae2\" (UID: \"8e28530d-6eb5-42ae-885d-5100c7d76ae2\") " Oct 31 00:02:00.350940 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:00.350924 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e28530d-6eb5-42ae-885d-5100c7d76ae2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e28530d-6eb5-42ae-885d-5100c7d76ae2-bundle\") pod \"8e28530d-6eb5-42ae-885d-5100c7d76ae2\" (UID: \"8e28530d-6eb5-42ae-885d-5100c7d76ae2\") " Oct 31 00:02:00.351130 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:00.351001 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e28530d-6eb5-42ae-885d-5100c7d76ae2-util\" (UniqueName: \"kubernetes.io/empty-dir/8e28530d-6eb5-42ae-885d-5100c7d76ae2-util\") pod \"8e28530d-6eb5-42ae-885d-5100c7d76ae2\" (UID: \"8e28530d-6eb5-42ae-885d-5100c7d76ae2\") " Oct 31 00:02:00.351539 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:00.351515 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e28530d-6eb5-42ae-885d-5100c7d76ae2-bundle" pod "8e28530d-6eb5-42ae-885d-5100c7d76ae2" (UID: "8e28530d-6eb5-42ae-885d-5100c7d76ae2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:02:00.353130 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:00.353104 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e28530d-6eb5-42ae-885d-5100c7d76ae2-kube-api-access-zrb79" pod "8e28530d-6eb5-42ae-885d-5100c7d76ae2" (UID: "8e28530d-6eb5-42ae-885d-5100c7d76ae2"). InnerVolumeSpecName "kube-api-access-zrb79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:02:00.355204 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:00.355185 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e28530d-6eb5-42ae-885d-5100c7d76ae2-util" pod "8e28530d-6eb5-42ae-885d-5100c7d76ae2" (UID: "8e28530d-6eb5-42ae-885d-5100c7d76ae2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:02:00.451810 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:00.451789 5274 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e28530d-6eb5-42ae-885d-5100c7d76ae2-util\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:00.451810 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:00.451811 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrb79\" (UniqueName: \"kubernetes.io/projected/8e28530d-6eb5-42ae-885d-5100c7d76ae2-kube-api-access-zrb79\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:00.451923 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:00.451820 5274 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e28530d-6eb5-42ae-885d-5100c7d76ae2-bundle\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:02:01.094314 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:01.094282 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpx4p" Oct 31 00:02:01.094534 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:01.094277 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmpx4p" event={"ID":"8e28530d-6eb5-42ae-885d-5100c7d76ae2","Type":"ContainerDied","Data":"3c0162efff32aa6f063641131d6028285e2f84d7c283e20cf8bde48c94d143ae"} Oct 31 00:02:01.094534 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:01.094407 5274 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c0162efff32aa6f063641131d6028285e2f84d7c283e20cf8bde48c94d143ae" Oct 31 00:02:05.542068 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:05.542034 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-mk7km"] Oct 31 00:02:05.542478 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:05.542299 5274 cpu_manager.go:414] "containerMap: removing container" podUID="8e28530d-6eb5-42ae-885d-5100c7d76ae2" containerName="extract" Oct 31 00:02:05.542478 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:05.542309 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e28530d-6eb5-42ae-885d-5100c7d76ae2" containerName="extract" Oct 31 00:02:05.542478 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:05.542326 5274 cpu_manager.go:414] "containerMap: removing container" podUID="8e28530d-6eb5-42ae-885d-5100c7d76ae2" containerName="pull" Oct 31 00:02:05.542478 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:05.542331 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e28530d-6eb5-42ae-885d-5100c7d76ae2" containerName="pull" Oct 31 00:02:05.542478 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:05.542347 5274 cpu_manager.go:414] "containerMap: removing container" podUID="8e28530d-6eb5-42ae-885d-5100c7d76ae2" containerName="util" Oct 31 00:02:05.542478 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:05.542357 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e28530d-6eb5-42ae-885d-5100c7d76ae2" containerName="util" Oct 31 00:02:05.542478 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:05.542398 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e28530d-6eb5-42ae-885d-5100c7d76ae2" containerName="extract" Oct 31 00:02:05.547207 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:05.545864 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-mk7km" Oct 31 00:02:05.550312 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:05.550290 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-6d6dp\"" Oct 31 00:02:05.551069 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:05.551049 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:02:05.551167 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:05.551150 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 31 00:02:05.560906 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:05.560880 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-mk7km"] Oct 31 00:02:05.690023 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:05.689994 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thvd9\" (UniqueName: \"kubernetes.io/projected/ae5af6d2-4b90-410b-b943-f0ad3b9b1e8e-kube-api-access-thvd9\") pod \"cert-manager-operator-controller-manager-65fc875df4-mk7km\" (UID: \"ae5af6d2-4b90-410b-b943-f0ad3b9b1e8e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-mk7km" Oct 31 00:02:05.690161 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:05.690105 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae5af6d2-4b90-410b-b943-f0ad3b9b1e8e-tmp\") pod \"cert-manager-operator-controller-manager-65fc875df4-mk7km\" (UID: \"ae5af6d2-4b90-410b-b943-f0ad3b9b1e8e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-mk7km" Oct 31 00:02:05.791155 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:05.791120 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae5af6d2-4b90-410b-b943-f0ad3b9b1e8e-tmp\") pod \"cert-manager-operator-controller-manager-65fc875df4-mk7km\" (UID: \"ae5af6d2-4b90-410b-b943-f0ad3b9b1e8e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-mk7km" Oct 31 00:02:05.791259 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:05.791163 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thvd9\" (UniqueName: \"kubernetes.io/projected/ae5af6d2-4b90-410b-b943-f0ad3b9b1e8e-kube-api-access-thvd9\") pod \"cert-manager-operator-controller-manager-65fc875df4-mk7km\" (UID: \"ae5af6d2-4b90-410b-b943-f0ad3b9b1e8e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-mk7km" Oct 31 00:02:05.791472 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:05.791454 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae5af6d2-4b90-410b-b943-f0ad3b9b1e8e-tmp\") pod \"cert-manager-operator-controller-manager-65fc875df4-mk7km\" (UID: \"ae5af6d2-4b90-410b-b943-f0ad3b9b1e8e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-mk7km" Oct 31 00:02:05.803564 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:05.803504 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thvd9\" (UniqueName: \"kubernetes.io/projected/ae5af6d2-4b90-410b-b943-f0ad3b9b1e8e-kube-api-access-thvd9\") pod \"cert-manager-operator-controller-manager-65fc875df4-mk7km\" (UID: \"ae5af6d2-4b90-410b-b943-f0ad3b9b1e8e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-mk7km" Oct 31 00:02:05.858286 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:05.858265 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-mk7km" Oct 31 00:02:06.133821 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:06.133797 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-mk7km"] Oct 31 00:02:06.136504 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:02:06.136477 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae5af6d2_4b90_410b_b943_f0ad3b9b1e8e.slice/crio-ec2c9bb16e9f0b8c61a7f28811b689cd2b7a9c7ccee860b5e37da7907ea22be7 WatchSource:0}: Error finding container ec2c9bb16e9f0b8c61a7f28811b689cd2b7a9c7ccee860b5e37da7907ea22be7: Status 404 returned error can't find the container with id ec2c9bb16e9f0b8c61a7f28811b689cd2b7a9c7ccee860b5e37da7907ea22be7 Oct 31 00:02:07.123366 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:07.123331 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-mk7km" event={"ID":"ae5af6d2-4b90-410b-b943-f0ad3b9b1e8e","Type":"ContainerStarted","Data":"ec2c9bb16e9f0b8c61a7f28811b689cd2b7a9c7ccee860b5e37da7907ea22be7"} Oct 31 00:02:09.135655 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:09.135617 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-mk7km" event={"ID":"ae5af6d2-4b90-410b-b943-f0ad3b9b1e8e","Type":"ContainerStarted","Data":"bc9cb60e8c92f41615df4b06cfe3fbaee10db0973f737748703aff9314776b7d"} Oct 31 00:02:09.168405 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:09.168354 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-mk7km" podStartSLOduration=1.827818015 podStartE2EDuration="4.168339799s" totalImagesPullingTime="2.340521784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:09.167017428 +0000 UTC m=+165.803836149" watchObservedRunningTime="2026-10-31 00:02:09.168339799 +0000 UTC m=+165.805158514" Oct 31 00:02:12.882914 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:12.882878 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-9twz7"] Oct 31 00:02:12.886087 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:12.886070 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9twz7" Oct 31 00:02:12.897544 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:12.897513 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-vb55p\"" Oct 31 00:02:12.899095 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:12.899070 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 31 00:02:12.899180 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:12.899169 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 31 00:02:12.899463 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:12.899446 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-9twz7"] Oct 31 00:02:13.045057 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:13.045028 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lgxc\" (UniqueName: \"kubernetes.io/projected/540908ca-7c84-4271-aee4-09fff5e2ebdd-kube-api-access-9lgxc\") pod \"cert-manager-cainjector-5c57bbd4fc-9twz7\" (UID: \"540908ca-7c84-4271-aee4-09fff5e2ebdd\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9twz7" Oct 31 00:02:13.045170 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:13.045069 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/540908ca-7c84-4271-aee4-09fff5e2ebdd-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-9twz7\" (UID: \"540908ca-7c84-4271-aee4-09fff5e2ebdd\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9twz7" Oct 31 00:02:13.146283 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:13.146227 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lgxc\" (UniqueName: \"kubernetes.io/projected/540908ca-7c84-4271-aee4-09fff5e2ebdd-kube-api-access-9lgxc\") pod \"cert-manager-cainjector-5c57bbd4fc-9twz7\" (UID: \"540908ca-7c84-4271-aee4-09fff5e2ebdd\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9twz7" Oct 31 00:02:13.146283 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:13.146266 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/540908ca-7c84-4271-aee4-09fff5e2ebdd-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-9twz7\" (UID: \"540908ca-7c84-4271-aee4-09fff5e2ebdd\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9twz7" Oct 31 00:02:13.159126 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:13.159102 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lgxc\" (UniqueName: \"kubernetes.io/projected/540908ca-7c84-4271-aee4-09fff5e2ebdd-kube-api-access-9lgxc\") pod \"cert-manager-cainjector-5c57bbd4fc-9twz7\" (UID: \"540908ca-7c84-4271-aee4-09fff5e2ebdd\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9twz7" Oct 31 00:02:13.159211 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:13.159106 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/540908ca-7c84-4271-aee4-09fff5e2ebdd-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-9twz7\" (UID: \"540908ca-7c84-4271-aee4-09fff5e2ebdd\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9twz7" Oct 31 00:02:13.208454 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:13.208435 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9twz7" Oct 31 00:02:13.480848 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:02:13.480813 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod540908ca_7c84_4271_aee4_09fff5e2ebdd.slice/crio-3581e23eed18f437e1db4294e813bd505d4222c5f28d5c4dd53f79655343188d WatchSource:0}: Error finding container 3581e23eed18f437e1db4294e813bd505d4222c5f28d5c4dd53f79655343188d: Status 404 returned error can't find the container with id 3581e23eed18f437e1db4294e813bd505d4222c5f28d5c4dd53f79655343188d Oct 31 00:02:13.481004 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:13.480984 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-9twz7"] Oct 31 00:02:14.156755 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:14.156699 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9twz7" event={"ID":"540908ca-7c84-4271-aee4-09fff5e2ebdd","Type":"ContainerStarted","Data":"3581e23eed18f437e1db4294e813bd505d4222c5f28d5c4dd53f79655343188d"} Oct 31 00:02:17.170332 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:17.170293 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9twz7" event={"ID":"540908ca-7c84-4271-aee4-09fff5e2ebdd","Type":"ContainerStarted","Data":"c881747f6009f48f07ce3608d03b0e90c27315cbc70cbe0f6ae1a8d13427daf4"} Oct 31 00:02:17.192571 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:17.192530 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9twz7" podStartSLOduration=2.492927826 podStartE2EDuration="5.1925176s" totalImagesPullingTime="2.699589774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:17.191266902 +0000 UTC m=+173.828085611" watchObservedRunningTime="2026-10-31 00:02:17.1925176 +0000 UTC m=+173.829336315" Oct 31 00:02:17.296906 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:17.296875 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-z6qqh"] Oct 31 00:02:17.300073 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:17.300054 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z6qqh" Oct 31 00:02:17.304147 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:17.304129 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-7vztt\"" Oct 31 00:02:17.312444 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:17.312424 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-z6qqh"] Oct 31 00:02:17.378486 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:17.378458 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eb122a05-4ec0-4214-b891-eba79cb32582-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-z6qqh\" (UID: \"eb122a05-4ec0-4214-b891-eba79cb32582\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z6qqh" Oct 31 00:02:17.378580 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:17.378506 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w42wt\" (UniqueName: \"kubernetes.io/projected/eb122a05-4ec0-4214-b891-eba79cb32582-kube-api-access-w42wt\") pod \"cert-manager-webhook-5dbbbf75cc-z6qqh\" (UID: \"eb122a05-4ec0-4214-b891-eba79cb32582\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z6qqh" Oct 31 00:02:17.479154 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:17.479094 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eb122a05-4ec0-4214-b891-eba79cb32582-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-z6qqh\" (UID: \"eb122a05-4ec0-4214-b891-eba79cb32582\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z6qqh" Oct 31 00:02:17.479154 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:17.479140 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w42wt\" (UniqueName: \"kubernetes.io/projected/eb122a05-4ec0-4214-b891-eba79cb32582-kube-api-access-w42wt\") pod \"cert-manager-webhook-5dbbbf75cc-z6qqh\" (UID: \"eb122a05-4ec0-4214-b891-eba79cb32582\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z6qqh" Oct 31 00:02:17.491386 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:17.491360 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eb122a05-4ec0-4214-b891-eba79cb32582-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-z6qqh\" (UID: \"eb122a05-4ec0-4214-b891-eba79cb32582\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z6qqh" Oct 31 00:02:17.491545 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:17.491525 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w42wt\" (UniqueName: \"kubernetes.io/projected/eb122a05-4ec0-4214-b891-eba79cb32582-kube-api-access-w42wt\") pod \"cert-manager-webhook-5dbbbf75cc-z6qqh\" (UID: \"eb122a05-4ec0-4214-b891-eba79cb32582\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z6qqh" Oct 31 00:02:17.611464 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:17.611438 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z6qqh" Oct 31 00:02:17.881195 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:17.881168 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-z6qqh"] Oct 31 00:02:17.884254 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:02:17.884220 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb122a05_4ec0_4214_b891_eba79cb32582.slice/crio-4657cbb285efa9f9286d8cde74219aeffb727ab7d3999bb9f4efbdbcff1782f8 WatchSource:0}: Error finding container 4657cbb285efa9f9286d8cde74219aeffb727ab7d3999bb9f4efbdbcff1782f8: Status 404 returned error can't find the container with id 4657cbb285efa9f9286d8cde74219aeffb727ab7d3999bb9f4efbdbcff1782f8 Oct 31 00:02:18.176625 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:18.176527 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z6qqh" event={"ID":"eb122a05-4ec0-4214-b891-eba79cb32582","Type":"ContainerStarted","Data":"008feee5f36ce84b739ffe0fbbd34286f7d292071dde42cbe45ee0ed49164a85"} Oct 31 00:02:18.176625 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:18.176580 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z6qqh" event={"ID":"eb122a05-4ec0-4214-b891-eba79cb32582","Type":"ContainerStarted","Data":"4657cbb285efa9f9286d8cde74219aeffb727ab7d3999bb9f4efbdbcff1782f8"} Oct 31 00:02:18.176625 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:18.176613 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z6qqh" Oct 31 00:02:18.195382 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:18.195344 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z6qqh" podStartSLOduration=1.195332598 podStartE2EDuration="1.195332598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:18.195163915 +0000 UTC m=+174.831982630" watchObservedRunningTime="2026-10-31 00:02:18.195332598 +0000 UTC m=+174.832151292" Oct 31 00:02:19.720255 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:19.720222 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-vb5tg"] Oct 31 00:02:19.723391 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:19.723371 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-vb5tg" Oct 31 00:02:19.728112 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:19.728094 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-qw7h4\"" Oct 31 00:02:19.733945 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:19.733923 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-vb5tg"] Oct 31 00:02:19.794321 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:19.794294 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n9s5\" (UniqueName: \"kubernetes.io/projected/937d1f7b-cbd6-404d-bda4-f98b46e27194-kube-api-access-8n9s5\") pod \"cert-manager-597b95cb84-vb5tg\" (UID: \"937d1f7b-cbd6-404d-bda4-f98b46e27194\") " pod="cert-manager/cert-manager-597b95cb84-vb5tg" Oct 31 00:02:19.794411 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:19.794383 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/937d1f7b-cbd6-404d-bda4-f98b46e27194-bound-sa-token\") pod \"cert-manager-597b95cb84-vb5tg\" (UID: \"937d1f7b-cbd6-404d-bda4-f98b46e27194\") " pod="cert-manager/cert-manager-597b95cb84-vb5tg" Oct 31 00:02:19.894760 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:19.894737 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/937d1f7b-cbd6-404d-bda4-f98b46e27194-bound-sa-token\") pod \"cert-manager-597b95cb84-vb5tg\" (UID: \"937d1f7b-cbd6-404d-bda4-f98b46e27194\") " pod="cert-manager/cert-manager-597b95cb84-vb5tg" Oct 31 00:02:19.894834 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:19.894771 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n9s5\" (UniqueName: \"kubernetes.io/projected/937d1f7b-cbd6-404d-bda4-f98b46e27194-kube-api-access-8n9s5\") pod \"cert-manager-597b95cb84-vb5tg\" (UID: \"937d1f7b-cbd6-404d-bda4-f98b46e27194\") " pod="cert-manager/cert-manager-597b95cb84-vb5tg" Oct 31 00:02:19.907468 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:19.907446 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/937d1f7b-cbd6-404d-bda4-f98b46e27194-bound-sa-token\") pod \"cert-manager-597b95cb84-vb5tg\" (UID: \"937d1f7b-cbd6-404d-bda4-f98b46e27194\") " pod="cert-manager/cert-manager-597b95cb84-vb5tg" Oct 31 00:02:19.907550 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:19.907524 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n9s5\" (UniqueName: \"kubernetes.io/projected/937d1f7b-cbd6-404d-bda4-f98b46e27194-kube-api-access-8n9s5\") pod \"cert-manager-597b95cb84-vb5tg\" (UID: \"937d1f7b-cbd6-404d-bda4-f98b46e27194\") " pod="cert-manager/cert-manager-597b95cb84-vb5tg" Oct 31 00:02:20.035086 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:20.035029 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-vb5tg" Oct 31 00:02:20.304801 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:20.304739 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-vb5tg"] Oct 31 00:02:20.307840 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:02:20.307815 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod937d1f7b_cbd6_404d_bda4_f98b46e27194.slice/crio-21dd9a38c9ed5ab4c496f64511d1c3293570d55f068193c0a6a272e6e74c1bdf WatchSource:0}: Error finding container 21dd9a38c9ed5ab4c496f64511d1c3293570d55f068193c0a6a272e6e74c1bdf: Status 404 returned error can't find the container with id 21dd9a38c9ed5ab4c496f64511d1c3293570d55f068193c0a6a272e6e74c1bdf Oct 31 00:02:20.898797 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:02:20.898764 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:02:21.195414 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:21.195332 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-vb5tg" event={"ID":"937d1f7b-cbd6-404d-bda4-f98b46e27194","Type":"ContainerStarted","Data":"731b22892d996c6d5267eaa532f45b9fe75647ac656d3fa9c865c880c141e3dc"} Oct 31 00:02:21.195414 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:21.195374 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-vb5tg" event={"ID":"937d1f7b-cbd6-404d-bda4-f98b46e27194","Type":"ContainerStarted","Data":"21dd9a38c9ed5ab4c496f64511d1c3293570d55f068193c0a6a272e6e74c1bdf"} Oct 31 00:02:21.213294 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:21.213251 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-vb5tg" podStartSLOduration=2.213235582 podStartE2EDuration="2.213235582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:21.2124486 +0000 UTC m=+177.849267316" watchObservedRunningTime="2026-10-31 00:02:21.213235582 +0000 UTC m=+177.850054310" Oct 31 00:02:24.183329 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:02:24.183295 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z6qqh" Oct 31 00:02:50.898391 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:02:50.898351 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:03:20.898643 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:03:20.898607 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:03:50.898187 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:03:50.898140 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:04:07.904352 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:07.904271 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054tffhx"] Oct 31 00:04:07.907703 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:07.907684 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054tffhx" Oct 31 00:04:07.912392 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:07.912374 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 31 00:04:07.912479 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:07.912396 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 31 00:04:07.912479 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:07.912445 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"07f0b9f07fd02dc1509b226b57404b9c198c3c-8adea440-dockercfg-kpzmk\"" Oct 31 00:04:07.917427 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:07.917407 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054tffhx"] Oct 31 00:04:08.046779 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:08.046753 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hhc4\" (UniqueName: \"kubernetes.io/projected/530635c2-5103-4796-87d4-8ce66a707317-kube-api-access-6hhc4\") pod \"07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054tffhx\" (UID: \"530635c2-5103-4796-87d4-8ce66a707317\") " pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054tffhx" Oct 31 00:04:08.046908 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:08.046785 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/530635c2-5103-4796-87d4-8ce66a707317-util\") pod \"07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054tffhx\" (UID: \"530635c2-5103-4796-87d4-8ce66a707317\") " pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054tffhx" Oct 31 00:04:08.046908 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:08.046809 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/530635c2-5103-4796-87d4-8ce66a707317-bundle\") pod \"07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054tffhx\" (UID: \"530635c2-5103-4796-87d4-8ce66a707317\") " pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054tffhx" Oct 31 00:04:08.148105 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:08.148077 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hhc4\" (UniqueName: \"kubernetes.io/projected/530635c2-5103-4796-87d4-8ce66a707317-kube-api-access-6hhc4\") pod \"07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054tffhx\" (UID: \"530635c2-5103-4796-87d4-8ce66a707317\") " pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054tffhx" Oct 31 00:04:08.148229 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:08.148109 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/530635c2-5103-4796-87d4-8ce66a707317-util\") pod \"07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054tffhx\" (UID: \"530635c2-5103-4796-87d4-8ce66a707317\") " pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054tffhx" Oct 31 00:04:08.148229 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:08.148136 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/530635c2-5103-4796-87d4-8ce66a707317-bundle\") pod \"07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054tffhx\" (UID: \"530635c2-5103-4796-87d4-8ce66a707317\") " pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054tffhx" Oct 31 00:04:08.148475 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:08.148461 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/530635c2-5103-4796-87d4-8ce66a707317-bundle\") pod \"07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054tffhx\" (UID: \"530635c2-5103-4796-87d4-8ce66a707317\") " pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054tffhx" Oct 31 00:04:08.148518 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:08.148494 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/530635c2-5103-4796-87d4-8ce66a707317-util\") pod \"07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054tffhx\" (UID: \"530635c2-5103-4796-87d4-8ce66a707317\") " pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054tffhx" Oct 31 00:04:08.158417 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:08.158362 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfb7qkg"] Oct 31 00:04:08.160715 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:08.160690 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hhc4\" (UniqueName: \"kubernetes.io/projected/530635c2-5103-4796-87d4-8ce66a707317-kube-api-access-6hhc4\") pod \"07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054tffhx\" (UID: \"530635c2-5103-4796-87d4-8ce66a707317\") " pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054tffhx" Oct 31 00:04:08.161702 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:08.161684 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfb7qkg" Oct 31 00:04:08.165600 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:08.165578 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6fb1a13d619f6ee73e221b33f4898cc831d48b-0ad9862f-dockercfg-ltkqx\"" Oct 31 00:04:08.171549 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:08.171496 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfb7qkg"] Oct 31 00:04:08.223334 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:08.223313 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054tffhx" Oct 31 00:04:08.249163 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:08.249140 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8563f96c-e65b-45eb-8931-c087d2ebfbcb-bundle\") pod \"6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfb7qkg\" (UID: \"8563f96c-e65b-45eb-8931-c087d2ebfbcb\") " pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfb7qkg" Oct 31 00:04:08.249265 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:08.249180 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdpfs\" (UniqueName: \"kubernetes.io/projected/8563f96c-e65b-45eb-8931-c087d2ebfbcb-kube-api-access-kdpfs\") pod \"6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfb7qkg\" (UID: \"8563f96c-e65b-45eb-8931-c087d2ebfbcb\") " pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfb7qkg" Oct 31 00:04:08.249265 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:08.249209 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8563f96c-e65b-45eb-8931-c087d2ebfbcb-util\") pod \"6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfb7qkg\" (UID: \"8563f96c-e65b-45eb-8931-c087d2ebfbcb\") " pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfb7qkg" Oct 31 00:04:08.349754 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:08.349722 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8563f96c-e65b-45eb-8931-c087d2ebfbcb-bundle\") pod \"6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfb7qkg\" (UID: \"8563f96c-e65b-45eb-8931-c087d2ebfbcb\") " pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfb7qkg" Oct 31 00:04:08.349895 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:08.349762 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdpfs\" (UniqueName: \"kubernetes.io/projected/8563f96c-e65b-45eb-8931-c087d2ebfbcb-kube-api-access-kdpfs\") pod \"6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfb7qkg\" (UID: \"8563f96c-e65b-45eb-8931-c087d2ebfbcb\") " pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfb7qkg" Oct 31 00:04:08.349895 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:08.349788 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8563f96c-e65b-45eb-8931-c087d2ebfbcb-util\") pod \"6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfb7qkg\" (UID: \"8563f96c-e65b-45eb-8931-c087d2ebfbcb\") " pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfb7qkg" Oct 31 00:04:08.350131 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:08.350113 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8563f96c-e65b-45eb-8931-c087d2ebfbcb-bundle\") pod \"6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfb7qkg\" (UID: \"8563f96c-e65b-45eb-8931-c087d2ebfbcb\") " pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfb7qkg" Oct 31 00:04:08.350168 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:08.350149 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8563f96c-e65b-45eb-8931-c087d2ebfbcb-util\") pod \"6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfb7qkg\" (UID: \"8563f96c-e65b-45eb-8931-c087d2ebfbcb\") " pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfb7qkg" Oct 31 00:04:08.362169 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:08.362141 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdpfs\" (UniqueName: \"kubernetes.io/projected/8563f96c-e65b-45eb-8931-c087d2ebfbcb-kube-api-access-kdpfs\") pod \"6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfb7qkg\" (UID: \"8563f96c-e65b-45eb-8931-c087d2ebfbcb\") " pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfb7qkg" Oct 31 00:04:08.482626 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:08.482576 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfb7qkg" Oct 31 00:04:08.494059 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:08.494032 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054tffhx"] Oct 31 00:04:08.497274 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:04:08.497253 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod530635c2_5103_4796_87d4_8ce66a707317.slice/crio-bb12879658f7cbc05a088764d9e14d28fcf1f1db9331d91bc283e4d1179b63ff WatchSource:0}: Error finding container bb12879658f7cbc05a088764d9e14d28fcf1f1db9331d91bc283e4d1179b63ff: Status 404 returned error can't find the container with id bb12879658f7cbc05a088764d9e14d28fcf1f1db9331d91bc283e4d1179b63ff Oct 31 00:04:08.604524 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:08.604499 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054tffhx" event={"ID":"530635c2-5103-4796-87d4-8ce66a707317","Type":"ContainerStarted","Data":"55a493d17dce26b886fe7f3a1d5675f0f480c81326c569cb09dcf65f89ca6862"} Oct 31 00:04:08.604643 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:08.604549 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054tffhx" event={"ID":"530635c2-5103-4796-87d4-8ce66a707317","Type":"ContainerStarted","Data":"bb12879658f7cbc05a088764d9e14d28fcf1f1db9331d91bc283e4d1179b63ff"} Oct 31 00:04:08.759407 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:08.759340 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfb7qkg"] Oct 31 00:04:08.763108 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:04:08.763082 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8563f96c_e65b_45eb_8931_c087d2ebfbcb.slice/crio-de4a3f4987ba4e628b891ef305b7b7356adc6e297e40050e3d14c2adc125d270 WatchSource:0}: Error finding container de4a3f4987ba4e628b891ef305b7b7356adc6e297e40050e3d14c2adc125d270: Status 404 returned error can't find the container with id de4a3f4987ba4e628b891ef305b7b7356adc6e297e40050e3d14c2adc125d270 Oct 31 00:04:09.609592 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:09.609560 5274 generic.go:356] "Generic (PLEG): container finished" podID="8563f96c-e65b-45eb-8931-c087d2ebfbcb" containerID="4d8d1b77d1a7490e3b258fdf2fef1ab9af63fc623eb5da22096ba5291b2b3f29" exitCode=0 Oct 31 00:04:09.610077 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:09.609656 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfb7qkg" event={"ID":"8563f96c-e65b-45eb-8931-c087d2ebfbcb","Type":"ContainerDied","Data":"4d8d1b77d1a7490e3b258fdf2fef1ab9af63fc623eb5da22096ba5291b2b3f29"} Oct 31 00:04:09.610077 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:09.609739 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfb7qkg" event={"ID":"8563f96c-e65b-45eb-8931-c087d2ebfbcb","Type":"ContainerStarted","Data":"de4a3f4987ba4e628b891ef305b7b7356adc6e297e40050e3d14c2adc125d270"} Oct 31 00:04:09.611403 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:09.611371 5274 generic.go:356] "Generic (PLEG): container finished" podID="530635c2-5103-4796-87d4-8ce66a707317" containerID="55a493d17dce26b886fe7f3a1d5675f0f480c81326c569cb09dcf65f89ca6862" exitCode=0 Oct 31 00:04:09.611519 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:09.611475 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054tffhx" event={"ID":"530635c2-5103-4796-87d4-8ce66a707317","Type":"ContainerDied","Data":"55a493d17dce26b886fe7f3a1d5675f0f480c81326c569cb09dcf65f89ca6862"} Oct 31 00:04:10.617515 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:10.617480 5274 generic.go:356] "Generic (PLEG): container finished" podID="530635c2-5103-4796-87d4-8ce66a707317" containerID="91a62e1453534121afb13d9ddde18e31b88bb1813b6d4153f9cfa8156dc01e14" exitCode=0 Oct 31 00:04:10.618005 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:10.617520 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054tffhx" event={"ID":"530635c2-5103-4796-87d4-8ce66a707317","Type":"ContainerDied","Data":"91a62e1453534121afb13d9ddde18e31b88bb1813b6d4153f9cfa8156dc01e14"} Oct 31 00:04:11.023881 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:11.023852 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974wpnpq"] Oct 31 00:04:11.027161 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:11.027144 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974wpnpq" Oct 31 00:04:11.031311 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:11.031293 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e1df189bb31fb8c419c1e848482afda4dccb51-08dd003f-dockercfg-kfxx5\"" Oct 31 00:04:11.038623 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:11.038604 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974wpnpq"] Oct 31 00:04:11.170591 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:11.170537 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27-bundle\") pod \"e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974wpnpq\" (UID: \"d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27\") " pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974wpnpq" Oct 31 00:04:11.170680 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:11.170588 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27-util\") pod \"e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974wpnpq\" (UID: \"d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27\") " pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974wpnpq" Oct 31 00:04:11.170680 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:11.170673 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw9wk\" (UniqueName: \"kubernetes.io/projected/d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27-kube-api-access-zw9wk\") pod \"e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974wpnpq\" (UID: \"d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27\") " pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974wpnpq" Oct 31 00:04:11.271842 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:11.271819 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw9wk\" (UniqueName: \"kubernetes.io/projected/d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27-kube-api-access-zw9wk\") pod \"e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974wpnpq\" (UID: \"d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27\") " pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974wpnpq" Oct 31 00:04:11.271928 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:11.271854 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27-bundle\") pod \"e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974wpnpq\" (UID: \"d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27\") " pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974wpnpq" Oct 31 00:04:11.271928 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:11.271889 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27-util\") pod \"e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974wpnpq\" (UID: \"d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27\") " pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974wpnpq" Oct 31 00:04:11.272209 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:11.272193 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27-util\") pod \"e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974wpnpq\" (UID: \"d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27\") " pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974wpnpq" Oct 31 00:04:11.272275 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:11.272251 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27-bundle\") pod \"e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974wpnpq\" (UID: \"d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27\") " pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974wpnpq" Oct 31 00:04:11.284777 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:11.284750 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw9wk\" (UniqueName: \"kubernetes.io/projected/d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27-kube-api-access-zw9wk\") pod \"e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974wpnpq\" (UID: \"d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27\") " pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974wpnpq" Oct 31 00:04:11.356300 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:11.356281 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974wpnpq" Oct 31 00:04:11.623186 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:11.623157 5274 generic.go:356] "Generic (PLEG): container finished" podID="8563f96c-e65b-45eb-8931-c087d2ebfbcb" containerID="6c0328860b6bd50878fd1692f3d974d9205803cedc515494505e4b363840c6f0" exitCode=0 Oct 31 00:04:11.623557 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:11.623227 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfb7qkg" event={"ID":"8563f96c-e65b-45eb-8931-c087d2ebfbcb","Type":"ContainerDied","Data":"6c0328860b6bd50878fd1692f3d974d9205803cedc515494505e4b363840c6f0"} Oct 31 00:04:11.625562 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:11.625543 5274 generic.go:356] "Generic (PLEG): container finished" podID="530635c2-5103-4796-87d4-8ce66a707317" containerID="222c41a8a103daf5c58c02355320085287c359e423a36959d357b5147a1e80cc" exitCode=0 Oct 31 00:04:11.625834 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:11.625669 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054tffhx" event={"ID":"530635c2-5103-4796-87d4-8ce66a707317","Type":"ContainerDied","Data":"222c41a8a103daf5c58c02355320085287c359e423a36959d357b5147a1e80cc"} Oct 31 00:04:11.627498 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:11.627478 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974wpnpq"] Oct 31 00:04:11.634380 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:04:11.634308 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b4196f_92f3_46e4_9ea3_e8bfc5c64e27.slice/crio-8df449a29642c9f3676e062daded0b41e637a0155ac504fc7416a592fc3c6b36 WatchSource:0}: Error finding container 8df449a29642c9f3676e062daded0b41e637a0155ac504fc7416a592fc3c6b36: Status 404 returned error can't find the container with id 8df449a29642c9f3676e062daded0b41e637a0155ac504fc7416a592fc3c6b36 Oct 31 00:04:12.632633 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:12.632595 5274 generic.go:356] "Generic (PLEG): container finished" podID="d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27" containerID="8adf56c33a52cf78e24c840c752bc630cd8ce24af4fef7471b1b30fea4d15737" exitCode=0 Oct 31 00:04:12.633155 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:12.632695 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974wpnpq" event={"ID":"d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27","Type":"ContainerDied","Data":"8adf56c33a52cf78e24c840c752bc630cd8ce24af4fef7471b1b30fea4d15737"} Oct 31 00:04:12.633155 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:12.632753 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974wpnpq" event={"ID":"d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27","Type":"ContainerStarted","Data":"8df449a29642c9f3676e062daded0b41e637a0155ac504fc7416a592fc3c6b36"} Oct 31 00:04:12.635422 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:12.635400 5274 generic.go:356] "Generic (PLEG): container finished" podID="8563f96c-e65b-45eb-8931-c087d2ebfbcb" containerID="f554bee25c90a42f4d164e1a06373b75705587c2636f6d93ce3cb700c79ee482" exitCode=0 Oct 31 00:04:12.635515 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:12.635490 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfb7qkg" event={"ID":"8563f96c-e65b-45eb-8931-c087d2ebfbcb","Type":"ContainerDied","Data":"f554bee25c90a42f4d164e1a06373b75705587c2636f6d93ce3cb700c79ee482"} Oct 31 00:04:12.761132 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:12.761109 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054tffhx" Oct 31 00:04:12.884833 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:12.884758 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/530635c2-5103-4796-87d4-8ce66a707317-util\" (UniqueName: \"kubernetes.io/empty-dir/530635c2-5103-4796-87d4-8ce66a707317-util\") pod \"530635c2-5103-4796-87d4-8ce66a707317\" (UID: \"530635c2-5103-4796-87d4-8ce66a707317\") " Oct 31 00:04:12.884833 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:12.884808 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/530635c2-5103-4796-87d4-8ce66a707317-bundle\" (UniqueName: \"kubernetes.io/empty-dir/530635c2-5103-4796-87d4-8ce66a707317-bundle\") pod \"530635c2-5103-4796-87d4-8ce66a707317\" (UID: \"530635c2-5103-4796-87d4-8ce66a707317\") " Oct 31 00:04:12.884833 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:12.884841 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/530635c2-5103-4796-87d4-8ce66a707317-kube-api-access-6hhc4\" (UniqueName: \"kubernetes.io/projected/530635c2-5103-4796-87d4-8ce66a707317-kube-api-access-6hhc4\") pod \"530635c2-5103-4796-87d4-8ce66a707317\" (UID: \"530635c2-5103-4796-87d4-8ce66a707317\") " Oct 31 00:04:12.885346 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:12.885324 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530635c2-5103-4796-87d4-8ce66a707317-bundle" pod "530635c2-5103-4796-87d4-8ce66a707317" (UID: "530635c2-5103-4796-87d4-8ce66a707317"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:04:12.886919 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:12.886898 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/530635c2-5103-4796-87d4-8ce66a707317-kube-api-access-6hhc4" pod "530635c2-5103-4796-87d4-8ce66a707317" (UID: "530635c2-5103-4796-87d4-8ce66a707317"). InnerVolumeSpecName "kube-api-access-6hhc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:04:12.890003 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:12.889948 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530635c2-5103-4796-87d4-8ce66a707317-util" pod "530635c2-5103-4796-87d4-8ce66a707317" (UID: "530635c2-5103-4796-87d4-8ce66a707317"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:04:12.985660 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:12.985637 5274 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/530635c2-5103-4796-87d4-8ce66a707317-bundle\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:04:12.985751 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:12.985662 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hhc4\" (UniqueName: \"kubernetes.io/projected/530635c2-5103-4796-87d4-8ce66a707317-kube-api-access-6hhc4\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:04:12.985751 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:12.985677 5274 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/530635c2-5103-4796-87d4-8ce66a707317-util\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:04:13.641207 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:13.641181 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054tffhx" Oct 31 00:04:13.641207 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:13.641198 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a10054tffhx" event={"ID":"530635c2-5103-4796-87d4-8ce66a707317","Type":"ContainerDied","Data":"bb12879658f7cbc05a088764d9e14d28fcf1f1db9331d91bc283e4d1179b63ff"} Oct 31 00:04:13.641608 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:13.641270 5274 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb12879658f7cbc05a088764d9e14d28fcf1f1db9331d91bc283e4d1179b63ff" Oct 31 00:04:13.760579 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:13.760557 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfb7qkg" Oct 31 00:04:13.893807 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:13.893782 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8563f96c-e65b-45eb-8931-c087d2ebfbcb-util\" (UniqueName: \"kubernetes.io/empty-dir/8563f96c-e65b-45eb-8931-c087d2ebfbcb-util\") pod \"8563f96c-e65b-45eb-8931-c087d2ebfbcb\" (UID: \"8563f96c-e65b-45eb-8931-c087d2ebfbcb\") " Oct 31 00:04:13.893972 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:13.893835 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8563f96c-e65b-45eb-8931-c087d2ebfbcb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8563f96c-e65b-45eb-8931-c087d2ebfbcb-bundle\") pod \"8563f96c-e65b-45eb-8931-c087d2ebfbcb\" (UID: \"8563f96c-e65b-45eb-8931-c087d2ebfbcb\") " Oct 31 00:04:13.893972 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:13.893875 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8563f96c-e65b-45eb-8931-c087d2ebfbcb-kube-api-access-kdpfs\" (UniqueName: \"kubernetes.io/projected/8563f96c-e65b-45eb-8931-c087d2ebfbcb-kube-api-access-kdpfs\") pod \"8563f96c-e65b-45eb-8931-c087d2ebfbcb\" (UID: \"8563f96c-e65b-45eb-8931-c087d2ebfbcb\") " Oct 31 00:04:13.895135 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:13.895093 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8563f96c-e65b-45eb-8931-c087d2ebfbcb-bundle" pod "8563f96c-e65b-45eb-8931-c087d2ebfbcb" (UID: "8563f96c-e65b-45eb-8931-c087d2ebfbcb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:04:13.895732 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:13.895704 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8563f96c-e65b-45eb-8931-c087d2ebfbcb-kube-api-access-kdpfs" pod "8563f96c-e65b-45eb-8931-c087d2ebfbcb" (UID: "8563f96c-e65b-45eb-8931-c087d2ebfbcb"). InnerVolumeSpecName "kube-api-access-kdpfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:04:13.963131 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:13.963106 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8563f96c-e65b-45eb-8931-c087d2ebfbcb-util" pod "8563f96c-e65b-45eb-8931-c087d2ebfbcb" (UID: "8563f96c-e65b-45eb-8931-c087d2ebfbcb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:04:13.995240 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:13.995220 5274 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8563f96c-e65b-45eb-8931-c087d2ebfbcb-bundle\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:04:13.995328 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:13.995243 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdpfs\" (UniqueName: \"kubernetes.io/projected/8563f96c-e65b-45eb-8931-c087d2ebfbcb-kube-api-access-kdpfs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:04:13.995328 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:13.995258 5274 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8563f96c-e65b-45eb-8931-c087d2ebfbcb-util\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:04:14.648158 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:14.648124 5274 generic.go:356] "Generic (PLEG): container finished" podID="d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27" containerID="3d23bf000e4bb63cce698adf43073496ec6546c6825aa99c3ef1da84974fd897" exitCode=0 Oct 31 00:04:14.648590 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:14.648182 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974wpnpq" event={"ID":"d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27","Type":"ContainerDied","Data":"3d23bf000e4bb63cce698adf43073496ec6546c6825aa99c3ef1da84974fd897"} Oct 31 00:04:14.650807 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:14.650784 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfb7qkg" event={"ID":"8563f96c-e65b-45eb-8931-c087d2ebfbcb","Type":"ContainerDied","Data":"de4a3f4987ba4e628b891ef305b7b7356adc6e297e40050e3d14c2adc125d270"} Oct 31 00:04:14.650807 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:14.650800 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfb7qkg" Oct 31 00:04:14.650972 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:14.650826 5274 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de4a3f4987ba4e628b891ef305b7b7356adc6e297e40050e3d14c2adc125d270" Oct 31 00:04:15.657348 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:15.657317 5274 generic.go:356] "Generic (PLEG): container finished" podID="d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27" containerID="3e9640de9a278fac5220ab5885a6badc29574c4f1ce548f5f9412525df0d1a0d" exitCode=0 Oct 31 00:04:15.657693 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:15.657370 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974wpnpq" event={"ID":"d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27","Type":"ContainerDied","Data":"3e9640de9a278fac5220ab5885a6badc29574c4f1ce548f5f9412525df0d1a0d"} Oct 31 00:04:16.779670 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:16.779647 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974wpnpq" Oct 31 00:04:16.920776 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:16.920714 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27-util\" (UniqueName: \"kubernetes.io/empty-dir/d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27-util\") pod \"d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27\" (UID: \"d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27\") " Oct 31 00:04:16.920881 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:16.920780 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27-bundle\") pod \"d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27\" (UID: \"d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27\") " Oct 31 00:04:16.920881 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:16.920810 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27-kube-api-access-zw9wk\" (UniqueName: \"kubernetes.io/projected/d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27-kube-api-access-zw9wk\") pod \"d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27\" (UID: \"d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27\") " Oct 31 00:04:16.921496 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:16.921470 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27-bundle" pod "d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27" (UID: "d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:04:16.922827 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:16.922805 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27-kube-api-access-zw9wk" pod "d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27" (UID: "d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27"). InnerVolumeSpecName "kube-api-access-zw9wk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:04:16.926118 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:16.926095 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27-util" pod "d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27" (UID: "d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:04:17.021429 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:17.021405 5274 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27-util\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:04:17.021429 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:17.021427 5274 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27-bundle\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:04:17.021547 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:17.021437 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw9wk\" (UniqueName: \"kubernetes.io/projected/d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27-kube-api-access-zw9wk\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:04:17.666374 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:17.666348 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974wpnpq" Oct 31 00:04:17.666374 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:17.666352 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e974wpnpq" event={"ID":"d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27","Type":"ContainerDied","Data":"8df449a29642c9f3676e062daded0b41e637a0155ac504fc7416a592fc3c6b36"} Oct 31 00:04:17.666580 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:17.666397 5274 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8df449a29642c9f3676e062daded0b41e637a0155ac504fc7416a592fc3c6b36" Oct 31 00:04:18.218071 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.218041 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-tempo-operator/tempo-operator-controller-c9df98c-ldhgb"] Oct 31 00:04:18.218440 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.218329 5274 cpu_manager.go:414] "containerMap: removing container" podUID="d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27" containerName="extract" Oct 31 00:04:18.218440 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.218339 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27" containerName="extract" Oct 31 00:04:18.218440 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.218349 5274 cpu_manager.go:414] "containerMap: removing container" podUID="8563f96c-e65b-45eb-8931-c087d2ebfbcb" containerName="util" Oct 31 00:04:18.218440 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.218354 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8563f96c-e65b-45eb-8931-c087d2ebfbcb" containerName="util" Oct 31 00:04:18.218440 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.218369 5274 cpu_manager.go:414] "containerMap: removing container" podUID="530635c2-5103-4796-87d4-8ce66a707317" containerName="extract" Oct 31 00:04:18.218440 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.218374 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="530635c2-5103-4796-87d4-8ce66a707317" containerName="extract" Oct 31 00:04:18.218440 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.218380 5274 cpu_manager.go:414] "containerMap: removing container" podUID="d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27" containerName="util" Oct 31 00:04:18.218440 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.218384 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27" containerName="util" Oct 31 00:04:18.218440 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.218399 5274 cpu_manager.go:414] "containerMap: removing container" podUID="530635c2-5103-4796-87d4-8ce66a707317" containerName="pull" Oct 31 00:04:18.218440 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.218403 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="530635c2-5103-4796-87d4-8ce66a707317" containerName="pull" Oct 31 00:04:18.218440 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.218409 5274 cpu_manager.go:414] "containerMap: removing container" podUID="8563f96c-e65b-45eb-8931-c087d2ebfbcb" containerName="extract" Oct 31 00:04:18.218440 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.218413 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8563f96c-e65b-45eb-8931-c087d2ebfbcb" containerName="extract" Oct 31 00:04:18.218440 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.218421 5274 cpu_manager.go:414] "containerMap: removing container" podUID="d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27" containerName="pull" Oct 31 00:04:18.218440 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.218425 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27" containerName="pull" Oct 31 00:04:18.218440 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.218433 5274 cpu_manager.go:414] "containerMap: removing container" podUID="8563f96c-e65b-45eb-8931-c087d2ebfbcb" containerName="pull" Oct 31 00:04:18.218440 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.218437 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8563f96c-e65b-45eb-8931-c087d2ebfbcb" containerName="pull" Oct 31 00:04:18.218440 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.218448 5274 cpu_manager.go:414] "containerMap: removing container" podUID="530635c2-5103-4796-87d4-8ce66a707317" containerName="util" Oct 31 00:04:18.218987 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.218452 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="530635c2-5103-4796-87d4-8ce66a707317" containerName="util" Oct 31 00:04:18.218987 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.218495 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="530635c2-5103-4796-87d4-8ce66a707317" containerName="extract" Oct 31 00:04:18.218987 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.218503 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="8563f96c-e65b-45eb-8931-c087d2ebfbcb" containerName="extract" Oct 31 00:04:18.218987 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.218511 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6b4196f-92f3-46e4-9ea3-e8bfc5c64e27" containerName="extract" Oct 31 00:04:18.226764 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.226738 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-tempo-operator/tempo-operator-controller-c9df98c-ldhgb" Oct 31 00:04:18.235356 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.235334 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-tempo-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:04:18.236026 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.235671 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-tempo-operator\"/\"webhook-server-cert\"" Oct 31 00:04:18.236026 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.235811 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-tempo-operator\"/\"kube-root-ca.crt\"" Oct 31 00:04:18.236026 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.235813 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-tempo-operator\"/\"tempo-operator-controller-manager-dockercfg-6zqfr\"" Oct 31 00:04:18.236026 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.235887 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-tempo-operator\"/\"tempo-operator-controller-service-cert\"" Oct 31 00:04:18.236026 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.235931 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-tempo-operator\"/\"tempo-operator-metrics\"" Oct 31 00:04:18.241610 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.241590 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-tempo-operator/tempo-operator-controller-c9df98c-ldhgb"] Oct 31 00:04:18.332577 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.332551 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2c734e0a-735a-4fee-aada-75fc9e68ee66-tempo-operator-metrics-cert\") pod \"tempo-operator-controller-c9df98c-ldhgb\" (UID: \"2c734e0a-735a-4fee-aada-75fc9e68ee66\") " pod="openshift-tempo-operator/tempo-operator-controller-c9df98c-ldhgb" Oct 31 00:04:18.332694 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.332598 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hntml\" (UniqueName: \"kubernetes.io/projected/2c734e0a-735a-4fee-aada-75fc9e68ee66-kube-api-access-hntml\") pod \"tempo-operator-controller-c9df98c-ldhgb\" (UID: \"2c734e0a-735a-4fee-aada-75fc9e68ee66\") " pod="openshift-tempo-operator/tempo-operator-controller-c9df98c-ldhgb" Oct 31 00:04:18.332694 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.332616 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2c734e0a-735a-4fee-aada-75fc9e68ee66-apiservice-cert\") pod \"tempo-operator-controller-c9df98c-ldhgb\" (UID: \"2c734e0a-735a-4fee-aada-75fc9e68ee66\") " pod="openshift-tempo-operator/tempo-operator-controller-c9df98c-ldhgb" Oct 31 00:04:18.332781 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.332729 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2c734e0a-735a-4fee-aada-75fc9e68ee66-webhook-cert\") pod \"tempo-operator-controller-c9df98c-ldhgb\" (UID: \"2c734e0a-735a-4fee-aada-75fc9e68ee66\") " pod="openshift-tempo-operator/tempo-operator-controller-c9df98c-ldhgb" Oct 31 00:04:18.433825 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.433796 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hntml\" (UniqueName: \"kubernetes.io/projected/2c734e0a-735a-4fee-aada-75fc9e68ee66-kube-api-access-hntml\") pod \"tempo-operator-controller-c9df98c-ldhgb\" (UID: \"2c734e0a-735a-4fee-aada-75fc9e68ee66\") " pod="openshift-tempo-operator/tempo-operator-controller-c9df98c-ldhgb" Oct 31 00:04:18.433925 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.433831 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2c734e0a-735a-4fee-aada-75fc9e68ee66-apiservice-cert\") pod \"tempo-operator-controller-c9df98c-ldhgb\" (UID: \"2c734e0a-735a-4fee-aada-75fc9e68ee66\") " pod="openshift-tempo-operator/tempo-operator-controller-c9df98c-ldhgb" Oct 31 00:04:18.433925 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.433852 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2c734e0a-735a-4fee-aada-75fc9e68ee66-webhook-cert\") pod \"tempo-operator-controller-c9df98c-ldhgb\" (UID: \"2c734e0a-735a-4fee-aada-75fc9e68ee66\") " pod="openshift-tempo-operator/tempo-operator-controller-c9df98c-ldhgb" Oct 31 00:04:18.434016 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.433923 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2c734e0a-735a-4fee-aada-75fc9e68ee66-tempo-operator-metrics-cert\") pod \"tempo-operator-controller-c9df98c-ldhgb\" (UID: \"2c734e0a-735a-4fee-aada-75fc9e68ee66\") " pod="openshift-tempo-operator/tempo-operator-controller-c9df98c-ldhgb" Oct 31 00:04:18.436269 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.436250 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2c734e0a-735a-4fee-aada-75fc9e68ee66-apiservice-cert\") pod \"tempo-operator-controller-c9df98c-ldhgb\" (UID: \"2c734e0a-735a-4fee-aada-75fc9e68ee66\") " pod="openshift-tempo-operator/tempo-operator-controller-c9df98c-ldhgb" Oct 31 00:04:18.436371 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.436336 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2c734e0a-735a-4fee-aada-75fc9e68ee66-webhook-cert\") pod \"tempo-operator-controller-c9df98c-ldhgb\" (UID: \"2c734e0a-735a-4fee-aada-75fc9e68ee66\") " pod="openshift-tempo-operator/tempo-operator-controller-c9df98c-ldhgb" Oct 31 00:04:18.436371 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.436335 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2c734e0a-735a-4fee-aada-75fc9e68ee66-tempo-operator-metrics-cert\") pod \"tempo-operator-controller-c9df98c-ldhgb\" (UID: \"2c734e0a-735a-4fee-aada-75fc9e68ee66\") " pod="openshift-tempo-operator/tempo-operator-controller-c9df98c-ldhgb" Oct 31 00:04:18.446180 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.446158 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hntml\" (UniqueName: \"kubernetes.io/projected/2c734e0a-735a-4fee-aada-75fc9e68ee66-kube-api-access-hntml\") pod \"tempo-operator-controller-c9df98c-ldhgb\" (UID: \"2c734e0a-735a-4fee-aada-75fc9e68ee66\") " pod="openshift-tempo-operator/tempo-operator-controller-c9df98c-ldhgb" Oct 31 00:04:18.540645 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.540591 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-tempo-operator/tempo-operator-controller-c9df98c-ldhgb" Oct 31 00:04:18.816003 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:18.815977 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-tempo-operator/tempo-operator-controller-c9df98c-ldhgb"] Oct 31 00:04:18.818394 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:04:18.818368 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c734e0a_735a_4fee_aada_75fc9e68ee66.slice/crio-e35402f9db3c8c4af88d188fa43454e69332fb0348a9acb67333840790f521ae WatchSource:0}: Error finding container e35402f9db3c8c4af88d188fa43454e69332fb0348a9acb67333840790f521ae: Status 404 returned error can't find the container with id e35402f9db3c8c4af88d188fa43454e69332fb0348a9acb67333840790f521ae Oct 31 00:04:19.675836 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:19.675801 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-tempo-operator/tempo-operator-controller-c9df98c-ldhgb" event={"ID":"2c734e0a-735a-4fee-aada-75fc9e68ee66","Type":"ContainerStarted","Data":"e35402f9db3c8c4af88d188fa43454e69332fb0348a9acb67333840790f521ae"} Oct 31 00:04:20.904422 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:04:20.904383 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:04:22.692127 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:22.692088 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-tempo-operator/tempo-operator-controller-c9df98c-ldhgb" event={"ID":"2c734e0a-735a-4fee-aada-75fc9e68ee66","Type":"ContainerStarted","Data":"4721f32d46e01d2885379aae53efee91b16912a23c0f039836248b7926e9109b"} Oct 31 00:04:22.692489 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:22.692227 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-tempo-operator/tempo-operator-controller-c9df98c-ldhgb" Oct 31 00:04:22.730039 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:22.729980 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-tempo-operator/tempo-operator-controller-c9df98c-ldhgb" podStartSLOduration=1.879998032 podStartE2EDuration="4.729939928s" totalImagesPullingTime="2.849941896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:04:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:04:22.728039067 +0000 UTC m=+299.364857784" watchObservedRunningTime="2026-10-31 00:04:22.729939928 +0000 UTC m=+299.366758646" Oct 31 00:04:32.236998 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.236944 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7fb7b74868-gwkrc"] Oct 31 00:04:32.245447 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.245414 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7fb7b74868-gwkrc" Oct 31 00:04:32.253851 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.253814 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"obo-prometheus-operator-admission-webhook-service-cert\"" Oct 31 00:04:32.253990 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.253932 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"obo-prometheus-operator-admission-webhook-dockercfg-fgczj\"" Oct 31 00:04:32.255696 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.255671 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7fb7b74868-gwkrc"] Oct 31 00:04:32.332664 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.332632 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/900d5ad4-111a-4a47-9809-643ab8000284-apiservice-cert\") pod \"obo-prometheus-operator-admission-webhook-7fb7b74868-gwkrc\" (UID: \"900d5ad4-111a-4a47-9809-643ab8000284\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7fb7b74868-gwkrc" Oct 31 00:04:32.332799 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.332747 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/900d5ad4-111a-4a47-9809-643ab8000284-webhook-cert\") pod \"obo-prometheus-operator-admission-webhook-7fb7b74868-gwkrc\" (UID: \"900d5ad4-111a-4a47-9809-643ab8000284\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7fb7b74868-gwkrc" Oct 31 00:04:32.433467 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.433418 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/900d5ad4-111a-4a47-9809-643ab8000284-webhook-cert\") pod \"obo-prometheus-operator-admission-webhook-7fb7b74868-gwkrc\" (UID: \"900d5ad4-111a-4a47-9809-643ab8000284\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7fb7b74868-gwkrc" Oct 31 00:04:32.433608 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.433538 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/900d5ad4-111a-4a47-9809-643ab8000284-apiservice-cert\") pod \"obo-prometheus-operator-admission-webhook-7fb7b74868-gwkrc\" (UID: \"900d5ad4-111a-4a47-9809-643ab8000284\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7fb7b74868-gwkrc" Oct 31 00:04:32.435785 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.435762 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/900d5ad4-111a-4a47-9809-643ab8000284-webhook-cert\") pod \"obo-prometheus-operator-admission-webhook-7fb7b74868-gwkrc\" (UID: \"900d5ad4-111a-4a47-9809-643ab8000284\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7fb7b74868-gwkrc" Oct 31 00:04:32.435881 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.435791 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/900d5ad4-111a-4a47-9809-643ab8000284-apiservice-cert\") pod \"obo-prometheus-operator-admission-webhook-7fb7b74868-gwkrc\" (UID: \"900d5ad4-111a-4a47-9809-643ab8000284\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7fb7b74868-gwkrc" Oct 31 00:04:32.560973 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.560899 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7fb7b74868-gwkrc" Oct 31 00:04:32.596043 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.596018 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-p6g75"] Oct 31 00:04:32.602437 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.602083 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-p6g75" Oct 31 00:04:32.614924 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.614381 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-opentelemetry-operator\"/\"kube-root-ca.crt\"" Oct 31 00:04:32.614924 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.614469 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-opentelemetry-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:04:32.614924 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.614644 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-opentelemetry-operator\"/\"opentelemetry-operator-controller-manager-dockercfg-dbkp4\"" Oct 31 00:04:32.614924 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.614749 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-opentelemetry-operator\"/\"opentelemetry-operator-metrics\"" Oct 31 00:04:32.626017 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.625946 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-p6g75"] Oct 31 00:04:32.635362 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.635339 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b4eaad8-f26b-4848-a9b9-f750079c11e0-opentelemetry-operator-metrics-cert\") pod \"opentelemetry-operator-controller-manager-5f695b7fbf-p6g75\" (UID: \"2b4eaad8-f26b-4848-a9b9-f750079c11e0\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-p6g75" Oct 31 00:04:32.635445 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.635370 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf9ss\" (UniqueName: \"kubernetes.io/projected/2b4eaad8-f26b-4848-a9b9-f750079c11e0-kube-api-access-kf9ss\") pod \"opentelemetry-operator-controller-manager-5f695b7fbf-p6g75\" (UID: \"2b4eaad8-f26b-4848-a9b9-f750079c11e0\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-p6g75" Oct 31 00:04:32.690924 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.690897 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-observability-operator/observability-operator-599c6f89d-bd595"] Oct 31 00:04:32.694186 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.694170 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-bd595" Oct 31 00:04:32.713788 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.713762 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-observability-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:04:32.714058 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.713879 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-observability-operator\"/\"kube-root-ca.crt\"" Oct 31 00:04:32.714755 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.714736 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"observability-operator-tls\"" Oct 31 00:04:32.714856 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.714775 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"observability-operator-sa-dockercfg-6mbtm\"" Oct 31 00:04:32.715698 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.715675 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/observability-operator-599c6f89d-bd595"] Oct 31 00:04:32.735944 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.735924 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b4eaad8-f26b-4848-a9b9-f750079c11e0-opentelemetry-operator-metrics-cert\") pod \"opentelemetry-operator-controller-manager-5f695b7fbf-p6g75\" (UID: \"2b4eaad8-f26b-4848-a9b9-f750079c11e0\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-p6g75" Oct 31 00:04:32.736045 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.735963 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf9ss\" (UniqueName: \"kubernetes.io/projected/2b4eaad8-f26b-4848-a9b9-f750079c11e0-kube-api-access-kf9ss\") pod \"opentelemetry-operator-controller-manager-5f695b7fbf-p6g75\" (UID: \"2b4eaad8-f26b-4848-a9b9-f750079c11e0\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-p6g75" Oct 31 00:04:32.736045 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.736004 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"observability-operator-tls\" (UniqueName: \"kubernetes.io/secret/f811b29c-8848-4fc6-a28c-91a034a55c0f-observability-operator-tls\") pod \"observability-operator-599c6f89d-bd595\" (UID: \"f811b29c-8848-4fc6-a28c-91a034a55c0f\") " pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-bd595" Oct 31 00:04:32.736140 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.736062 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wmm8\" (UniqueName: \"kubernetes.io/projected/f811b29c-8848-4fc6-a28c-91a034a55c0f-kube-api-access-9wmm8\") pod \"observability-operator-599c6f89d-bd595\" (UID: \"f811b29c-8848-4fc6-a28c-91a034a55c0f\") " pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-bd595" Oct 31 00:04:32.738299 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.738277 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2b4eaad8-f26b-4848-a9b9-f750079c11e0-opentelemetry-operator-metrics-cert\") pod \"opentelemetry-operator-controller-manager-5f695b7fbf-p6g75\" (UID: \"2b4eaad8-f26b-4848-a9b9-f750079c11e0\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-p6g75" Oct 31 00:04:32.761926 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.761905 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf9ss\" (UniqueName: \"kubernetes.io/projected/2b4eaad8-f26b-4848-a9b9-f750079c11e0-kube-api-access-kf9ss\") pod \"opentelemetry-operator-controller-manager-5f695b7fbf-p6g75\" (UID: \"2b4eaad8-f26b-4848-a9b9-f750079c11e0\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-p6g75" Oct 31 00:04:32.836990 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.836931 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"observability-operator-tls\" (UniqueName: \"kubernetes.io/secret/f811b29c-8848-4fc6-a28c-91a034a55c0f-observability-operator-tls\") pod \"observability-operator-599c6f89d-bd595\" (UID: \"f811b29c-8848-4fc6-a28c-91a034a55c0f\") " pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-bd595" Oct 31 00:04:32.837071 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.837028 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wmm8\" (UniqueName: \"kubernetes.io/projected/f811b29c-8848-4fc6-a28c-91a034a55c0f-kube-api-access-9wmm8\") pod \"observability-operator-599c6f89d-bd595\" (UID: \"f811b29c-8848-4fc6-a28c-91a034a55c0f\") " pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-bd595" Oct 31 00:04:32.839057 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.839037 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"observability-operator-tls\" (UniqueName: \"kubernetes.io/secret/f811b29c-8848-4fc6-a28c-91a034a55c0f-observability-operator-tls\") pod \"observability-operator-599c6f89d-bd595\" (UID: \"f811b29c-8848-4fc6-a28c-91a034a55c0f\") " pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-bd595" Oct 31 00:04:32.846250 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.846210 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7fb7b74868-gwkrc"] Oct 31 00:04:32.849166 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:04:32.849145 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod900d5ad4_111a_4a47_9809_643ab8000284.slice/crio-f1ffad7ebf0418ce3f6c7a135d37acb488320e8c78e171316b868c5eacdde4b2 WatchSource:0}: Error finding container f1ffad7ebf0418ce3f6c7a135d37acb488320e8c78e171316b868c5eacdde4b2: Status 404 returned error can't find the container with id f1ffad7ebf0418ce3f6c7a135d37acb488320e8c78e171316b868c5eacdde4b2 Oct 31 00:04:32.855523 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.855503 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wmm8\" (UniqueName: \"kubernetes.io/projected/f811b29c-8848-4fc6-a28c-91a034a55c0f-kube-api-access-9wmm8\") pod \"observability-operator-599c6f89d-bd595\" (UID: \"f811b29c-8848-4fc6-a28c-91a034a55c0f\") " pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-bd595" Oct 31 00:04:32.918987 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:32.918965 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-p6g75" Oct 31 00:04:33.006520 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.006496 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-bd595" Oct 31 00:04:33.195551 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.195530 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-p6g75"] Oct 31 00:04:33.198041 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:04:33.198016 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b4eaad8_f26b_4848_a9b9_f750079c11e0.slice/crio-2b48279fdce9f1f7d760462bece396cb382f4a5a97df8b8ac95e376cbee9d99f WatchSource:0}: Error finding container 2b48279fdce9f1f7d760462bece396cb382f4a5a97df8b8ac95e376cbee9d99f: Status 404 returned error can't find the container with id 2b48279fdce9f1f7d760462bece396cb382f4a5a97df8b8ac95e376cbee9d99f Oct 31 00:04:33.296210 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.296184 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/observability-operator-599c6f89d-bd595"] Oct 31 00:04:33.299481 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:04:33.299458 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf811b29c_8848_4fc6_a28c_91a034a55c0f.slice/crio-4d703ce60ae6b53655ce6683a55214432d49a33274a8d7cc57bfc6e9ffacc036 WatchSource:0}: Error finding container 4d703ce60ae6b53655ce6683a55214432d49a33274a8d7cc57bfc6e9ffacc036: Status 404 returned error can't find the container with id 4d703ce60ae6b53655ce6683a55214432d49a33274a8d7cc57bfc6e9ffacc036 Oct 31 00:04:33.302602 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.302581 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-observability-operator/perses-operator-7c856d4d4c-x67rf"] Oct 31 00:04:33.307293 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.307277 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/perses-operator-7c856d4d4c-x67rf" Oct 31 00:04:33.316530 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.316510 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"perses-operator-dockercfg-26whr\"" Oct 31 00:04:33.316631 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.316567 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"perses-operator-service-cert\"" Oct 31 00:04:33.320413 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.320392 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/perses-operator-7c856d4d4c-x67rf"] Oct 31 00:04:33.342222 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.342201 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4d969e19-4f85-4e6f-9c5a-e373c53a7bde-webhook-cert\") pod \"perses-operator-7c856d4d4c-x67rf\" (UID: \"4d969e19-4f85-4e6f-9c5a-e373c53a7bde\") " pod="openshift-cluster-observability-operator/perses-operator-7c856d4d4c-x67rf" Oct 31 00:04:33.342346 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.342234 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d969e19-4f85-4e6f-9c5a-e373c53a7bde-openshift-service-ca\") pod \"perses-operator-7c856d4d4c-x67rf\" (UID: \"4d969e19-4f85-4e6f-9c5a-e373c53a7bde\") " pod="openshift-cluster-observability-operator/perses-operator-7c856d4d4c-x67rf" Oct 31 00:04:33.342346 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.342267 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4d969e19-4f85-4e6f-9c5a-e373c53a7bde-apiservice-cert\") pod \"perses-operator-7c856d4d4c-x67rf\" (UID: \"4d969e19-4f85-4e6f-9c5a-e373c53a7bde\") " pod="openshift-cluster-observability-operator/perses-operator-7c856d4d4c-x67rf" Oct 31 00:04:33.342346 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.342319 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkttq\" (UniqueName: \"kubernetes.io/projected/4d969e19-4f85-4e6f-9c5a-e373c53a7bde-kube-api-access-qkttq\") pod \"perses-operator-7c856d4d4c-x67rf\" (UID: \"4d969e19-4f85-4e6f-9c5a-e373c53a7bde\") " pod="openshift-cluster-observability-operator/perses-operator-7c856d4d4c-x67rf" Oct 31 00:04:33.443634 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.443608 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d969e19-4f85-4e6f-9c5a-e373c53a7bde-openshift-service-ca\") pod \"perses-operator-7c856d4d4c-x67rf\" (UID: \"4d969e19-4f85-4e6f-9c5a-e373c53a7bde\") " pod="openshift-cluster-observability-operator/perses-operator-7c856d4d4c-x67rf" Oct 31 00:04:33.443729 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.443662 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4d969e19-4f85-4e6f-9c5a-e373c53a7bde-apiservice-cert\") pod \"perses-operator-7c856d4d4c-x67rf\" (UID: \"4d969e19-4f85-4e6f-9c5a-e373c53a7bde\") " pod="openshift-cluster-observability-operator/perses-operator-7c856d4d4c-x67rf" Oct 31 00:04:33.443729 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.443689 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkttq\" (UniqueName: \"kubernetes.io/projected/4d969e19-4f85-4e6f-9c5a-e373c53a7bde-kube-api-access-qkttq\") pod \"perses-operator-7c856d4d4c-x67rf\" (UID: \"4d969e19-4f85-4e6f-9c5a-e373c53a7bde\") " pod="openshift-cluster-observability-operator/perses-operator-7c856d4d4c-x67rf" Oct 31 00:04:33.443854 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.443766 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4d969e19-4f85-4e6f-9c5a-e373c53a7bde-webhook-cert\") pod \"perses-operator-7c856d4d4c-x67rf\" (UID: \"4d969e19-4f85-4e6f-9c5a-e373c53a7bde\") " pod="openshift-cluster-observability-operator/perses-operator-7c856d4d4c-x67rf" Oct 31 00:04:33.444361 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.444341 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d969e19-4f85-4e6f-9c5a-e373c53a7bde-openshift-service-ca\") pod \"perses-operator-7c856d4d4c-x67rf\" (UID: \"4d969e19-4f85-4e6f-9c5a-e373c53a7bde\") " pod="openshift-cluster-observability-operator/perses-operator-7c856d4d4c-x67rf" Oct 31 00:04:33.446112 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.446069 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4d969e19-4f85-4e6f-9c5a-e373c53a7bde-apiservice-cert\") pod \"perses-operator-7c856d4d4c-x67rf\" (UID: \"4d969e19-4f85-4e6f-9c5a-e373c53a7bde\") " pod="openshift-cluster-observability-operator/perses-operator-7c856d4d4c-x67rf" Oct 31 00:04:33.446239 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.446218 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4d969e19-4f85-4e6f-9c5a-e373c53a7bde-webhook-cert\") pod \"perses-operator-7c856d4d4c-x67rf\" (UID: \"4d969e19-4f85-4e6f-9c5a-e373c53a7bde\") " pod="openshift-cluster-observability-operator/perses-operator-7c856d4d4c-x67rf" Oct 31 00:04:33.462905 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.462887 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkttq\" (UniqueName: \"kubernetes.io/projected/4d969e19-4f85-4e6f-9c5a-e373c53a7bde-kube-api-access-qkttq\") pod \"perses-operator-7c856d4d4c-x67rf\" (UID: \"4d969e19-4f85-4e6f-9c5a-e373c53a7bde\") " pod="openshift-cluster-observability-operator/perses-operator-7c856d4d4c-x67rf" Oct 31 00:04:33.596662 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.596636 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-6c7d65b66-cxcqb"] Oct 31 00:04:33.600040 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.600025 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-6c7d65b66-cxcqb" Oct 31 00:04:33.607395 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.607322 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-opentelemetry-operator\"/\"opentelemetry-operator-webhook-service-cert\"" Oct 31 00:04:33.616049 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.616027 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-6c7d65b66-cxcqb"] Oct 31 00:04:33.619805 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.619788 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/perses-operator-7c856d4d4c-x67rf" Oct 31 00:04:33.646115 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.646087 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c86982f7-6d7d-463d-a107-033c44b29306-apiservice-cert\") pod \"opentelemetry-operator-webhook-6c7d65b66-cxcqb\" (UID: \"c86982f7-6d7d-463d-a107-033c44b29306\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-6c7d65b66-cxcqb" Oct 31 00:04:33.646323 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.646290 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c86982f7-6d7d-463d-a107-033c44b29306-webhook-cert\") pod \"opentelemetry-operator-webhook-6c7d65b66-cxcqb\" (UID: \"c86982f7-6d7d-463d-a107-033c44b29306\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-6c7d65b66-cxcqb" Oct 31 00:04:33.646434 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.646367 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgd5k\" (UniqueName: \"kubernetes.io/projected/c86982f7-6d7d-463d-a107-033c44b29306-kube-api-access-bgd5k\") pod \"opentelemetry-operator-webhook-6c7d65b66-cxcqb\" (UID: \"c86982f7-6d7d-463d-a107-033c44b29306\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-6c7d65b66-cxcqb" Oct 31 00:04:33.699359 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.699305 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-tempo-operator/tempo-operator-controller-c9df98c-ldhgb" Oct 31 00:04:33.741835 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.741801 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-p6g75" event={"ID":"2b4eaad8-f26b-4848-a9b9-f750079c11e0","Type":"ContainerStarted","Data":"2b48279fdce9f1f7d760462bece396cb382f4a5a97df8b8ac95e376cbee9d99f"} Oct 31 00:04:33.743511 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.743484 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-bd595" event={"ID":"f811b29c-8848-4fc6-a28c-91a034a55c0f","Type":"ContainerStarted","Data":"4d703ce60ae6b53655ce6683a55214432d49a33274a8d7cc57bfc6e9ffacc036"} Oct 31 00:04:33.744933 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.744912 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7fb7b74868-gwkrc" event={"ID":"900d5ad4-111a-4a47-9809-643ab8000284","Type":"ContainerStarted","Data":"f1ffad7ebf0418ce3f6c7a135d37acb488320e8c78e171316b868c5eacdde4b2"} Oct 31 00:04:33.747311 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.747287 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c86982f7-6d7d-463d-a107-033c44b29306-apiservice-cert\") pod \"opentelemetry-operator-webhook-6c7d65b66-cxcqb\" (UID: \"c86982f7-6d7d-463d-a107-033c44b29306\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-6c7d65b66-cxcqb" Oct 31 00:04:33.747413 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.747339 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c86982f7-6d7d-463d-a107-033c44b29306-webhook-cert\") pod \"opentelemetry-operator-webhook-6c7d65b66-cxcqb\" (UID: \"c86982f7-6d7d-463d-a107-033c44b29306\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-6c7d65b66-cxcqb" Oct 31 00:04:33.747870 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.747579 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgd5k\" (UniqueName: \"kubernetes.io/projected/c86982f7-6d7d-463d-a107-033c44b29306-kube-api-access-bgd5k\") pod \"opentelemetry-operator-webhook-6c7d65b66-cxcqb\" (UID: \"c86982f7-6d7d-463d-a107-033c44b29306\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-6c7d65b66-cxcqb" Oct 31 00:04:33.749826 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.749807 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c86982f7-6d7d-463d-a107-033c44b29306-webhook-cert\") pod \"opentelemetry-operator-webhook-6c7d65b66-cxcqb\" (UID: \"c86982f7-6d7d-463d-a107-033c44b29306\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-6c7d65b66-cxcqb" Oct 31 00:04:33.750394 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.750374 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c86982f7-6d7d-463d-a107-033c44b29306-apiservice-cert\") pod \"opentelemetry-operator-webhook-6c7d65b66-cxcqb\" (UID: \"c86982f7-6d7d-463d-a107-033c44b29306\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-6c7d65b66-cxcqb" Oct 31 00:04:33.760369 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.760347 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgd5k\" (UniqueName: \"kubernetes.io/projected/c86982f7-6d7d-463d-a107-033c44b29306-kube-api-access-bgd5k\") pod \"opentelemetry-operator-webhook-6c7d65b66-cxcqb\" (UID: \"c86982f7-6d7d-463d-a107-033c44b29306\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-6c7d65b66-cxcqb" Oct 31 00:04:33.906446 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.906421 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/perses-operator-7c856d4d4c-x67rf"] Oct 31 00:04:33.909945 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:04:33.909904 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d969e19_4f85_4e6f_9c5a_e373c53a7bde.slice/crio-73559cbc5fd10886aabfed9f87ddd68173f68cae500a9f6160fa011bd5385367 WatchSource:0}: Error finding container 73559cbc5fd10886aabfed9f87ddd68173f68cae500a9f6160fa011bd5385367: Status 404 returned error can't find the container with id 73559cbc5fd10886aabfed9f87ddd68173f68cae500a9f6160fa011bd5385367 Oct 31 00:04:33.911398 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:33.911371 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-6c7d65b66-cxcqb" Oct 31 00:04:34.194926 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:34.194897 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-6c7d65b66-cxcqb"] Oct 31 00:04:34.196688 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:04:34.196655 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc86982f7_6d7d_463d_a107_033c44b29306.slice/crio-5780787b804d862ca5e31c86ce445772bef15cd60fd899298b9906c0df50f9ee WatchSource:0}: Error finding container 5780787b804d862ca5e31c86ce445772bef15cd60fd899298b9906c0df50f9ee: Status 404 returned error can't find the container with id 5780787b804d862ca5e31c86ce445772bef15cd60fd899298b9906c0df50f9ee Oct 31 00:04:34.752807 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:34.752761 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-6c7d65b66-cxcqb" event={"ID":"c86982f7-6d7d-463d-a107-033c44b29306","Type":"ContainerStarted","Data":"5780787b804d862ca5e31c86ce445772bef15cd60fd899298b9906c0df50f9ee"} Oct 31 00:04:34.756536 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:34.756506 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/perses-operator-7c856d4d4c-x67rf" event={"ID":"4d969e19-4f85-4e6f-9c5a-e373c53a7bde","Type":"ContainerStarted","Data":"73559cbc5fd10886aabfed9f87ddd68173f68cae500a9f6160fa011bd5385367"} Oct 31 00:04:35.764654 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:35.764522 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-6c7d65b66-cxcqb" event={"ID":"c86982f7-6d7d-463d-a107-033c44b29306","Type":"ContainerStarted","Data":"0bf4d6ac1e65d34715a0cafc1e85631531ec75c4bcd2cd97465dba25bd463a2d"} Oct 31 00:04:35.764654 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:35.764618 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-6c7d65b66-cxcqb" Oct 31 00:04:35.766597 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:35.766568 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-p6g75" event={"ID":"2b4eaad8-f26b-4848-a9b9-f750079c11e0","Type":"ContainerStarted","Data":"3302418d8020684f66a4cdf6a4caffff7077406e746d7b298e834cb8c1bb8467"} Oct 31 00:04:35.766732 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:35.766625 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-p6g75" Oct 31 00:04:35.796405 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:35.796359 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-6c7d65b66-cxcqb" podStartSLOduration=1.397422312 podStartE2EDuration="2.796344877s" totalImagesPullingTime="1.398922565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:04:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:04:35.793828129 +0000 UTC m=+312.430646861" watchObservedRunningTime="2026-10-31 00:04:35.796344877 +0000 UTC m=+312.433163593" Oct 31 00:04:35.818514 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:35.818464 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-p6g75" podStartSLOduration=1.56937238 podStartE2EDuration="3.81845176s" totalImagesPullingTime="2.24907938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:04:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:04:35.818181927 +0000 UTC m=+312.455000645" watchObservedRunningTime="2026-10-31 00:04:35.81845176 +0000 UTC m=+312.455270476" Oct 31 00:04:36.772620 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:36.772526 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7fb7b74868-gwkrc" event={"ID":"900d5ad4-111a-4a47-9809-643ab8000284","Type":"ContainerStarted","Data":"810db44a854c8d0efa1f1a62978e7c632c7152a5ec7eab351dce4c4122588b62"} Oct 31 00:04:36.799548 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:36.799503 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7fb7b74868-gwkrc" podStartSLOduration=1.280636973 podStartE2EDuration="4.799491013s" totalImagesPullingTime="3.51885404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:04:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:04:36.797999566 +0000 UTC m=+313.434818282" watchObservedRunningTime="2026-10-31 00:04:36.799491013 +0000 UTC m=+313.436309728" Oct 31 00:04:39.788870 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:39.788831 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/perses-operator-7c856d4d4c-x67rf" event={"ID":"4d969e19-4f85-4e6f-9c5a-e373c53a7bde","Type":"ContainerStarted","Data":"17433e6bbec2c9e801189c0be17d9c2db5a5f73955c7de78011941c3ec1b71f2"} Oct 31 00:04:39.789317 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:39.789018 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-cluster-observability-operator/perses-operator-7c856d4d4c-x67rf" Oct 31 00:04:39.813782 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:39.813719 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-observability-operator/perses-operator-7c856d4d4c-x67rf" podStartSLOduration=1.920475119 podStartE2EDuration="6.813705449s" totalImagesPullingTime="4.89323033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:04:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:04:39.811227235 +0000 UTC m=+316.448045951" watchObservedRunningTime="2026-10-31 00:04:39.813705449 +0000 UTC m=+316.450524164" Oct 31 00:04:42.805538 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:42.805503 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-bd595" event={"ID":"f811b29c-8848-4fc6-a28c-91a034a55c0f","Type":"ContainerStarted","Data":"c02368ab3227da92ce19d230403b723a56327ff6f01a69ea324297ffccc47db3"} Oct 31 00:04:42.806014 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:42.805659 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-bd595" Oct 31 00:04:42.806597 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:42.806563 5274 patch_prober.go:29] interesting pod/observability-operator-599c6f89d-bd595 container/operator namespace/openshift-cluster-observability-operator: Readiness probe status=failure output="Get \"http://10.134.0.26:8081/healthz\": dial tcp 10.134.0.26:8081: connect: connection refused" start-of-body= Oct 31 00:04:42.806692 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:42.806624 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-bd595" podUID="f811b29c-8848-4fc6-a28c-91a034a55c0f" containerName="operator" probeResult="failure" output="Get \"http://10.134.0.26:8081/healthz\": dial tcp 10.134.0.26:8081: connect: connection refused" Oct 31 00:04:42.841345 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:42.841301 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-bd595" podStartSLOduration=1.466714746 podStartE2EDuration="10.841288065s" totalImagesPullingTime="9.374573319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:04:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:04:42.840586312 +0000 UTC m=+319.477405040" watchObservedRunningTime="2026-10-31 00:04:42.841288065 +0000 UTC m=+319.478106780" Oct 31 00:04:43.815697 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:43.815669 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-cluster-observability-operator/observability-operator-599c6f89d-bd595" Oct 31 00:04:46.774695 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:46.774668 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-p6g75" Oct 31 00:04:46.776105 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:46.776087 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-6c7d65b66-cxcqb" Oct 31 00:04:50.795658 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:50.795628 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-cluster-observability-operator/perses-operator-7c856d4d4c-x67rf" Oct 31 00:04:50.900108 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:04:50.900075 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:04:55.235984 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:55.235940 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadclksh"] Oct 31 00:04:55.239687 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:55.239667 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadclksh" Oct 31 00:04:55.246720 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:55.246697 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 31 00:04:55.246841 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:55.246737 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 31 00:04:55.246841 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:55.246765 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"d28b81ce69fab5e58d2cd73462dde4041bbc20-487d35f4-dockercfg-b6xgm\"" Oct 31 00:04:55.258564 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:55.258541 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadclksh"] Oct 31 00:04:55.338445 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:55.338417 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/507f43e7-e2bb-4f5d-b554-4d5b35aae424-bundle\") pod \"d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadclksh\" (UID: \"507f43e7-e2bb-4f5d-b554-4d5b35aae424\") " pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadclksh" Oct 31 00:04:55.338445 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:55.338446 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/507f43e7-e2bb-4f5d-b554-4d5b35aae424-util\") pod \"d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadclksh\" (UID: \"507f43e7-e2bb-4f5d-b554-4d5b35aae424\") " pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadclksh" Oct 31 00:04:55.338671 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:55.338470 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwm8n\" (UniqueName: \"kubernetes.io/projected/507f43e7-e2bb-4f5d-b554-4d5b35aae424-kube-api-access-dwm8n\") pod \"d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadclksh\" (UID: \"507f43e7-e2bb-4f5d-b554-4d5b35aae424\") " pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadclksh" Oct 31 00:04:55.439815 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:55.439787 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/507f43e7-e2bb-4f5d-b554-4d5b35aae424-bundle\") pod \"d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadclksh\" (UID: \"507f43e7-e2bb-4f5d-b554-4d5b35aae424\") " pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadclksh" Oct 31 00:04:55.439815 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:55.439815 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/507f43e7-e2bb-4f5d-b554-4d5b35aae424-util\") pod \"d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadclksh\" (UID: \"507f43e7-e2bb-4f5d-b554-4d5b35aae424\") " pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadclksh" Oct 31 00:04:55.440041 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:55.439836 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwm8n\" (UniqueName: \"kubernetes.io/projected/507f43e7-e2bb-4f5d-b554-4d5b35aae424-kube-api-access-dwm8n\") pod \"d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadclksh\" (UID: \"507f43e7-e2bb-4f5d-b554-4d5b35aae424\") " pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadclksh" Oct 31 00:04:55.440209 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:55.440189 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/507f43e7-e2bb-4f5d-b554-4d5b35aae424-util\") pod \"d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadclksh\" (UID: \"507f43e7-e2bb-4f5d-b554-4d5b35aae424\") " pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadclksh" Oct 31 00:04:55.440284 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:55.440251 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/507f43e7-e2bb-4f5d-b554-4d5b35aae424-bundle\") pod \"d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadclksh\" (UID: \"507f43e7-e2bb-4f5d-b554-4d5b35aae424\") " pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadclksh" Oct 31 00:04:55.455356 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:55.455333 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwm8n\" (UniqueName: \"kubernetes.io/projected/507f43e7-e2bb-4f5d-b554-4d5b35aae424-kube-api-access-dwm8n\") pod \"d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadclksh\" (UID: \"507f43e7-e2bb-4f5d-b554-4d5b35aae424\") " pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadclksh" Oct 31 00:04:55.555166 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:55.555098 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadclksh" Oct 31 00:04:55.829618 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:55.829580 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadclksh"] Oct 31 00:04:55.832639 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:04:55.832583 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod507f43e7_e2bb_4f5d_b554_4d5b35aae424.slice/crio-7d6a73ecbe48e5fdc3070d68265b31bd8734f03dbb5092b5a5ce48d986a936a4 WatchSource:0}: Error finding container 7d6a73ecbe48e5fdc3070d68265b31bd8734f03dbb5092b5a5ce48d986a936a4: Status 404 returned error can't find the container with id 7d6a73ecbe48e5fdc3070d68265b31bd8734f03dbb5092b5a5ce48d986a936a4 Oct 31 00:04:55.865013 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:55.864982 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadclksh" event={"ID":"507f43e7-e2bb-4f5d-b554-4d5b35aae424","Type":"ContainerStarted","Data":"7d6a73ecbe48e5fdc3070d68265b31bd8734f03dbb5092b5a5ce48d986a936a4"} Oct 31 00:04:56.871095 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:56.871058 5274 generic.go:356] "Generic (PLEG): container finished" podID="507f43e7-e2bb-4f5d-b554-4d5b35aae424" containerID="71a77ae79956068644e9b22c3f00ba6205aec020d3450c7311dfd637993f5a47" exitCode=0 Oct 31 00:04:56.871541 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:56.871114 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadclksh" event={"ID":"507f43e7-e2bb-4f5d-b554-4d5b35aae424","Type":"ContainerDied","Data":"71a77ae79956068644e9b22c3f00ba6205aec020d3450c7311dfd637993f5a47"} Oct 31 00:04:56.872361 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:56.872341 5274 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:04:58.882895 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:58.882857 5274 generic.go:356] "Generic (PLEG): container finished" podID="507f43e7-e2bb-4f5d-b554-4d5b35aae424" containerID="b0aca0313d6e7bedee99846e6581a01a3394cbda3e6ad942e8aa6e5e55826592" exitCode=0 Oct 31 00:04:58.883293 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:58.882939 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadclksh" event={"ID":"507f43e7-e2bb-4f5d-b554-4d5b35aae424","Type":"ContainerDied","Data":"b0aca0313d6e7bedee99846e6581a01a3394cbda3e6ad942e8aa6e5e55826592"} Oct 31 00:04:59.889673 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:59.889640 5274 generic.go:356] "Generic (PLEG): container finished" podID="507f43e7-e2bb-4f5d-b554-4d5b35aae424" containerID="ebea9c74e1c8252e3619b463361f171cd0e6a5947c319310973cebf2c272a65b" exitCode=0 Oct 31 00:04:59.890130 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:04:59.889725 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadclksh" event={"ID":"507f43e7-e2bb-4f5d-b554-4d5b35aae424","Type":"ContainerDied","Data":"ebea9c74e1c8252e3619b463361f171cd0e6a5947c319310973cebf2c272a65b"} Oct 31 00:05:01.015509 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:01.015486 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadclksh" Oct 31 00:05:01.084602 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:01.084575 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/507f43e7-e2bb-4f5d-b554-4d5b35aae424-kube-api-access-dwm8n\" (UniqueName: \"kubernetes.io/projected/507f43e7-e2bb-4f5d-b554-4d5b35aae424-kube-api-access-dwm8n\") pod \"507f43e7-e2bb-4f5d-b554-4d5b35aae424\" (UID: \"507f43e7-e2bb-4f5d-b554-4d5b35aae424\") " Oct 31 00:05:01.084741 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:01.084665 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/507f43e7-e2bb-4f5d-b554-4d5b35aae424-bundle\" (UniqueName: \"kubernetes.io/empty-dir/507f43e7-e2bb-4f5d-b554-4d5b35aae424-bundle\") pod \"507f43e7-e2bb-4f5d-b554-4d5b35aae424\" (UID: \"507f43e7-e2bb-4f5d-b554-4d5b35aae424\") " Oct 31 00:05:01.084741 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:01.084694 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/507f43e7-e2bb-4f5d-b554-4d5b35aae424-util\" (UniqueName: \"kubernetes.io/empty-dir/507f43e7-e2bb-4f5d-b554-4d5b35aae424-util\") pod \"507f43e7-e2bb-4f5d-b554-4d5b35aae424\" (UID: \"507f43e7-e2bb-4f5d-b554-4d5b35aae424\") " Oct 31 00:05:01.085281 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:01.085256 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/507f43e7-e2bb-4f5d-b554-4d5b35aae424-bundle" pod "507f43e7-e2bb-4f5d-b554-4d5b35aae424" (UID: "507f43e7-e2bb-4f5d-b554-4d5b35aae424"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:05:01.086614 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:01.086592 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/507f43e7-e2bb-4f5d-b554-4d5b35aae424-kube-api-access-dwm8n" pod "507f43e7-e2bb-4f5d-b554-4d5b35aae424" (UID: "507f43e7-e2bb-4f5d-b554-4d5b35aae424"). InnerVolumeSpecName "kube-api-access-dwm8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:05:01.091580 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:01.091534 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/507f43e7-e2bb-4f5d-b554-4d5b35aae424-util" pod "507f43e7-e2bb-4f5d-b554-4d5b35aae424" (UID: "507f43e7-e2bb-4f5d-b554-4d5b35aae424"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:05:01.186342 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:01.186272 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwm8n\" (UniqueName: \"kubernetes.io/projected/507f43e7-e2bb-4f5d-b554-4d5b35aae424-kube-api-access-dwm8n\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:01.186342 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:01.186299 5274 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/507f43e7-e2bb-4f5d-b554-4d5b35aae424-bundle\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:01.186342 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:01.186310 5274 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/507f43e7-e2bb-4f5d-b554-4d5b35aae424-util\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:01.900208 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:01.900178 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadclksh" Oct 31 00:05:01.900208 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:01.900192 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadclksh" event={"ID":"507f43e7-e2bb-4f5d-b554-4d5b35aae424","Type":"ContainerDied","Data":"7d6a73ecbe48e5fdc3070d68265b31bd8734f03dbb5092b5a5ce48d986a936a4"} Oct 31 00:05:01.900459 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:01.900232 5274 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d6a73ecbe48e5fdc3070d68265b31bd8734f03dbb5092b5a5ce48d986a936a4" Oct 31 00:05:20.898912 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:05:20.898878 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:05:40.706588 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:40.706511 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722fspvr"] Oct 31 00:05:40.706971 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:40.706926 5274 cpu_manager.go:414] "containerMap: removing container" podUID="507f43e7-e2bb-4f5d-b554-4d5b35aae424" containerName="pull" Oct 31 00:05:40.706971 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:40.706942 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="507f43e7-e2bb-4f5d-b554-4d5b35aae424" containerName="pull" Oct 31 00:05:40.706971 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:40.706971 5274 cpu_manager.go:414] "containerMap: removing container" podUID="507f43e7-e2bb-4f5d-b554-4d5b35aae424" containerName="util" Oct 31 00:05:40.707091 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:40.706976 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="507f43e7-e2bb-4f5d-b554-4d5b35aae424" containerName="util" Oct 31 00:05:40.707091 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:40.706990 5274 cpu_manager.go:414] "containerMap: removing container" podUID="507f43e7-e2bb-4f5d-b554-4d5b35aae424" containerName="extract" Oct 31 00:05:40.707091 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:40.706996 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="507f43e7-e2bb-4f5d-b554-4d5b35aae424" containerName="extract" Oct 31 00:05:40.707091 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:40.707054 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="507f43e7-e2bb-4f5d-b554-4d5b35aae424" containerName="extract" Oct 31 00:05:40.710238 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:40.710220 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722fspvr" Oct 31 00:05:40.714361 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:40.714340 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 31 00:05:40.715560 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:40.715539 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 31 00:05:40.715560 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:40.715552 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"b60fd1021875ce556623478c7c5d4d22148dff-4f0c4bdd-dockercfg-pzgtt\"" Oct 31 00:05:40.720260 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:40.720239 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722fspvr"] Oct 31 00:05:40.806253 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:40.806226 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d1f8fce-c878-46f6-8e0f-bfff68ecff4d-bundle\") pod \"b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722fspvr\" (UID: \"0d1f8fce-c878-46f6-8e0f-bfff68ecff4d\") " pod="openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722fspvr" Oct 31 00:05:40.806253 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:40.806253 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d1f8fce-c878-46f6-8e0f-bfff68ecff4d-util\") pod \"b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722fspvr\" (UID: \"0d1f8fce-c878-46f6-8e0f-bfff68ecff4d\") " pod="openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722fspvr" Oct 31 00:05:40.806412 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:40.806281 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch7lb\" (UniqueName: \"kubernetes.io/projected/0d1f8fce-c878-46f6-8e0f-bfff68ecff4d-kube-api-access-ch7lb\") pod \"b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722fspvr\" (UID: \"0d1f8fce-c878-46f6-8e0f-bfff68ecff4d\") " pod="openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722fspvr" Oct 31 00:05:40.906733 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:40.906708 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch7lb\" (UniqueName: \"kubernetes.io/projected/0d1f8fce-c878-46f6-8e0f-bfff68ecff4d-kube-api-access-ch7lb\") pod \"b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722fspvr\" (UID: \"0d1f8fce-c878-46f6-8e0f-bfff68ecff4d\") " pod="openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722fspvr" Oct 31 00:05:40.906839 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:40.906770 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d1f8fce-c878-46f6-8e0f-bfff68ecff4d-bundle\") pod \"b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722fspvr\" (UID: \"0d1f8fce-c878-46f6-8e0f-bfff68ecff4d\") " pod="openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722fspvr" Oct 31 00:05:40.906839 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:40.906789 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d1f8fce-c878-46f6-8e0f-bfff68ecff4d-util\") pod \"b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722fspvr\" (UID: \"0d1f8fce-c878-46f6-8e0f-bfff68ecff4d\") " pod="openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722fspvr" Oct 31 00:05:40.907098 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:40.907084 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d1f8fce-c878-46f6-8e0f-bfff68ecff4d-util\") pod \"b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722fspvr\" (UID: \"0d1f8fce-c878-46f6-8e0f-bfff68ecff4d\") " pod="openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722fspvr" Oct 31 00:05:40.907164 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:40.907146 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d1f8fce-c878-46f6-8e0f-bfff68ecff4d-bundle\") pod \"b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722fspvr\" (UID: \"0d1f8fce-c878-46f6-8e0f-bfff68ecff4d\") " pod="openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722fspvr" Oct 31 00:05:40.920151 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:40.920126 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch7lb\" (UniqueName: \"kubernetes.io/projected/0d1f8fce-c878-46f6-8e0f-bfff68ecff4d-kube-api-access-ch7lb\") pod \"b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722fspvr\" (UID: \"0d1f8fce-c878-46f6-8e0f-bfff68ecff4d\") " pod="openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722fspvr" Oct 31 00:05:41.027338 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:41.027289 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722fspvr" Oct 31 00:05:41.300806 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:41.300780 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722fspvr"] Oct 31 00:05:41.303331 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:05:41.303305 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d1f8fce_c878_46f6_8e0f_bfff68ecff4d.slice/crio-b107333f659a80b61821445772fb5b07e18bed6b5e19c00963ec969212094fd7 WatchSource:0}: Error finding container b107333f659a80b61821445772fb5b07e18bed6b5e19c00963ec969212094fd7: Status 404 returned error can't find the container with id b107333f659a80b61821445772fb5b07e18bed6b5e19c00963ec969212094fd7 Oct 31 00:05:42.096771 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:42.096737 5274 generic.go:356] "Generic (PLEG): container finished" podID="0d1f8fce-c878-46f6-8e0f-bfff68ecff4d" containerID="be1384cc27480262bc7048293714050f5dd2d2b186801568c38e4860722b3855" exitCode=0 Oct 31 00:05:42.097191 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:42.096828 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722fspvr" event={"ID":"0d1f8fce-c878-46f6-8e0f-bfff68ecff4d","Type":"ContainerDied","Data":"be1384cc27480262bc7048293714050f5dd2d2b186801568c38e4860722b3855"} Oct 31 00:05:42.097191 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:42.096892 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722fspvr" event={"ID":"0d1f8fce-c878-46f6-8e0f-bfff68ecff4d","Type":"ContainerStarted","Data":"b107333f659a80b61821445772fb5b07e18bed6b5e19c00963ec969212094fd7"} Oct 31 00:05:44.107705 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:44.107629 5274 generic.go:356] "Generic (PLEG): container finished" podID="0d1f8fce-c878-46f6-8e0f-bfff68ecff4d" containerID="10eae406186c81304f60171ee795f620b62390995f2f5bf85697b2b159e61752" exitCode=0 Oct 31 00:05:44.108075 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:44.107715 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722fspvr" event={"ID":"0d1f8fce-c878-46f6-8e0f-bfff68ecff4d","Type":"ContainerDied","Data":"10eae406186c81304f60171ee795f620b62390995f2f5bf85697b2b159e61752"} Oct 31 00:05:45.115201 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:45.115167 5274 generic.go:356] "Generic (PLEG): container finished" podID="0d1f8fce-c878-46f6-8e0f-bfff68ecff4d" containerID="22702cceb51c3279702bd605bb3a2a03123e036c50e6288c4fda756a302ffbd2" exitCode=0 Oct 31 00:05:45.115626 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:45.115264 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722fspvr" event={"ID":"0d1f8fce-c878-46f6-8e0f-bfff68ecff4d","Type":"ContainerDied","Data":"22702cceb51c3279702bd605bb3a2a03123e036c50e6288c4fda756a302ffbd2"} Oct 31 00:05:46.238839 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:46.238816 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722fspvr" Oct 31 00:05:46.348401 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:46.348374 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d1f8fce-c878-46f6-8e0f-bfff68ecff4d-kube-api-access-ch7lb\" (UniqueName: \"kubernetes.io/projected/0d1f8fce-c878-46f6-8e0f-bfff68ecff4d-kube-api-access-ch7lb\") pod \"0d1f8fce-c878-46f6-8e0f-bfff68ecff4d\" (UID: \"0d1f8fce-c878-46f6-8e0f-bfff68ecff4d\") " Oct 31 00:05:46.348536 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:46.348462 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d1f8fce-c878-46f6-8e0f-bfff68ecff4d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d1f8fce-c878-46f6-8e0f-bfff68ecff4d-bundle\") pod \"0d1f8fce-c878-46f6-8e0f-bfff68ecff4d\" (UID: \"0d1f8fce-c878-46f6-8e0f-bfff68ecff4d\") " Oct 31 00:05:46.348536 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:46.348483 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d1f8fce-c878-46f6-8e0f-bfff68ecff4d-util\" (UniqueName: \"kubernetes.io/empty-dir/0d1f8fce-c878-46f6-8e0f-bfff68ecff4d-util\") pod \"0d1f8fce-c878-46f6-8e0f-bfff68ecff4d\" (UID: \"0d1f8fce-c878-46f6-8e0f-bfff68ecff4d\") " Oct 31 00:05:46.349052 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:46.349024 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d1f8fce-c878-46f6-8e0f-bfff68ecff4d-bundle" pod "0d1f8fce-c878-46f6-8e0f-bfff68ecff4d" (UID: "0d1f8fce-c878-46f6-8e0f-bfff68ecff4d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:05:46.350472 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:46.350446 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d1f8fce-c878-46f6-8e0f-bfff68ecff4d-kube-api-access-ch7lb" pod "0d1f8fce-c878-46f6-8e0f-bfff68ecff4d" (UID: "0d1f8fce-c878-46f6-8e0f-bfff68ecff4d"). InnerVolumeSpecName "kube-api-access-ch7lb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:05:46.353614 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:46.353574 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d1f8fce-c878-46f6-8e0f-bfff68ecff4d-util" pod "0d1f8fce-c878-46f6-8e0f-bfff68ecff4d" (UID: "0d1f8fce-c878-46f6-8e0f-bfff68ecff4d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:05:46.448931 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:46.448877 5274 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d1f8fce-c878-46f6-8e0f-bfff68ecff4d-bundle\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:46.448931 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:46.448898 5274 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d1f8fce-c878-46f6-8e0f-bfff68ecff4d-util\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:46.448931 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:46.448908 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch7lb\" (UniqueName: \"kubernetes.io/projected/0d1f8fce-c878-46f6-8e0f-bfff68ecff4d-kube-api-access-ch7lb\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:05:47.127472 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:47.127427 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722fspvr" event={"ID":"0d1f8fce-c878-46f6-8e0f-bfff68ecff4d","Type":"ContainerDied","Data":"b107333f659a80b61821445772fb5b07e18bed6b5e19c00963ec969212094fd7"} Oct 31 00:05:47.127472 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:47.127475 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722fspvr" Oct 31 00:05:47.127715 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:05:47.127495 5274 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b107333f659a80b61821445772fb5b07e18bed6b5e19c00963ec969212094fd7" Oct 31 00:05:50.900204 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:05:50.900161 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:06:16.369323 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:16.369289 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-ccl7x"] Oct 31 00:06:16.369838 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:16.369812 5274 cpu_manager.go:414] "containerMap: removing container" podUID="0d1f8fce-c878-46f6-8e0f-bfff68ecff4d" containerName="pull" Oct 31 00:06:16.369838 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:16.369833 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d1f8fce-c878-46f6-8e0f-bfff68ecff4d" containerName="pull" Oct 31 00:06:16.369981 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:16.369856 5274 cpu_manager.go:414] "containerMap: removing container" podUID="0d1f8fce-c878-46f6-8e0f-bfff68ecff4d" containerName="extract" Oct 31 00:06:16.369981 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:16.369864 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d1f8fce-c878-46f6-8e0f-bfff68ecff4d" containerName="extract" Oct 31 00:06:16.369981 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:16.369877 5274 cpu_manager.go:414] "containerMap: removing container" podUID="0d1f8fce-c878-46f6-8e0f-bfff68ecff4d" containerName="util" Oct 31 00:06:16.369981 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:16.369885 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d1f8fce-c878-46f6-8e0f-bfff68ecff4d" containerName="util" Oct 31 00:06:16.370171 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:16.370007 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d1f8fce-c878-46f6-8e0f-bfff68ecff4d" containerName="extract" Oct 31 00:06:16.373041 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:16.373021 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-ccl7x" Oct 31 00:06:16.377744 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:16.377725 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:06:16.378561 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:16.378541 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 31 00:06:16.378654 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:16.378561 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 31 00:06:16.378654 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:16.378585 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:06:16.378654 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:16.378647 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-zkff8\"" Oct 31 00:06:16.383596 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:16.383574 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-ccl7x"] Oct 31 00:06:16.486134 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:16.486103 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f1630767-8bfc-46fe-b014-d05cef42acae-config\") pod \"korrel8r-6644769df5-ccl7x\" (UID: \"f1630767-8bfc-46fe-b014-d05cef42acae\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-ccl7x" Oct 31 00:06:16.486288 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:16.486223 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f1630767-8bfc-46fe-b014-d05cef42acae-tls\") pod \"korrel8r-6644769df5-ccl7x\" (UID: \"f1630767-8bfc-46fe-b014-d05cef42acae\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-ccl7x" Oct 31 00:06:16.486288 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:16.486266 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbjlw\" (UniqueName: \"kubernetes.io/projected/f1630767-8bfc-46fe-b014-d05cef42acae-kube-api-access-jbjlw\") pod \"korrel8r-6644769df5-ccl7x\" (UID: \"f1630767-8bfc-46fe-b014-d05cef42acae\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-ccl7x" Oct 31 00:06:16.587640 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:16.587609 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbjlw\" (UniqueName: \"kubernetes.io/projected/f1630767-8bfc-46fe-b014-d05cef42acae-kube-api-access-jbjlw\") pod \"korrel8r-6644769df5-ccl7x\" (UID: \"f1630767-8bfc-46fe-b014-d05cef42acae\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-ccl7x" Oct 31 00:06:16.587802 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:16.587650 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f1630767-8bfc-46fe-b014-d05cef42acae-config\") pod \"korrel8r-6644769df5-ccl7x\" (UID: \"f1630767-8bfc-46fe-b014-d05cef42acae\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-ccl7x" Oct 31 00:06:16.587853 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:16.587831 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f1630767-8bfc-46fe-b014-d05cef42acae-tls\") pod \"korrel8r-6644769df5-ccl7x\" (UID: \"f1630767-8bfc-46fe-b014-d05cef42acae\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-ccl7x" Oct 31 00:06:16.588395 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:16.588372 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f1630767-8bfc-46fe-b014-d05cef42acae-config\") pod \"korrel8r-6644769df5-ccl7x\" (UID: \"f1630767-8bfc-46fe-b014-d05cef42acae\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-ccl7x" Oct 31 00:06:16.590209 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:16.590183 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f1630767-8bfc-46fe-b014-d05cef42acae-tls\") pod \"korrel8r-6644769df5-ccl7x\" (UID: \"f1630767-8bfc-46fe-b014-d05cef42acae\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-ccl7x" Oct 31 00:06:16.604901 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:16.604879 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbjlw\" (UniqueName: \"kubernetes.io/projected/f1630767-8bfc-46fe-b014-d05cef42acae-kube-api-access-jbjlw\") pod \"korrel8r-6644769df5-ccl7x\" (UID: \"f1630767-8bfc-46fe-b014-d05cef42acae\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-ccl7x" Oct 31 00:06:16.686392 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:16.686318 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-ccl7x" Oct 31 00:06:16.966805 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:16.966781 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-ccl7x"] Oct 31 00:06:16.969490 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:06:16.969454 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1630767_8bfc_46fe_b014_d05cef42acae.slice/crio-bf4b0774bf942fc692e5aef92b83d892d06c74b7ebbaf68d94f039887514c1cf WatchSource:0}: Error finding container bf4b0774bf942fc692e5aef92b83d892d06c74b7ebbaf68d94f039887514c1cf: Status 404 returned error can't find the container with id bf4b0774bf942fc692e5aef92b83d892d06c74b7ebbaf68d94f039887514c1cf Oct 31 00:06:17.271876 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:17.271788 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-ccl7x" event={"ID":"f1630767-8bfc-46fe-b014-d05cef42acae","Type":"ContainerStarted","Data":"bf4b0774bf942fc692e5aef92b83d892d06c74b7ebbaf68d94f039887514c1cf"} Oct 31 00:06:20.824348 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:20.824317 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:06:20.830028 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:20.830000 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:20.835615 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:20.835589 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 31 00:06:20.835615 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:20.835590 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\"" Oct 31 00:06:20.836646 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:20.836624 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\"" Oct 31 00:06:20.836646 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:20.836638 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 31 00:06:20.836844 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:20.836671 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-7d7qm\"" Oct 31 00:06:20.836844 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:20.836686 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\"" Oct 31 00:06:20.836844 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:20.836696 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\"" Oct 31 00:06:20.836844 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:20.836703 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\"" Oct 31 00:06:20.843140 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:20.843116 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:06:20.901668 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:06:20.901625 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:06:20.925324 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:20.925298 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/4e7a6b98-3c61-4cde-acee-900d239aa617-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:20.925484 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:20.925365 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e7a6b98-3c61-4cde-acee-900d239aa617-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:20.925484 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:20.925468 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/4e7a6b98-3c61-4cde-acee-900d239aa617-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:20.925603 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:20.925496 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/4e7a6b98-3c61-4cde-acee-900d239aa617-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:20.925603 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:20.925517 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxs6n\" (UniqueName: \"kubernetes.io/projected/4e7a6b98-3c61-4cde-acee-900d239aa617-kube-api-access-rxs6n\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:20.925712 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:20.925618 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e7a6b98-3c61-4cde-acee-900d239aa617-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:20.925712 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:20.925674 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/4e7a6b98-3c61-4cde-acee-900d239aa617-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:20.925815 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:20.925765 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/4e7a6b98-3c61-4cde-acee-900d239aa617-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:21.026500 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:21.026466 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e7a6b98-3c61-4cde-acee-900d239aa617-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:21.026675 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:21.026506 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/4e7a6b98-3c61-4cde-acee-900d239aa617-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:21.026675 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:21.026539 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/4e7a6b98-3c61-4cde-acee-900d239aa617-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:21.026675 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:21.026578 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/4e7a6b98-3c61-4cde-acee-900d239aa617-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:21.026675 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:21.026629 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e7a6b98-3c61-4cde-acee-900d239aa617-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:21.026675 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:21.026670 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/4e7a6b98-3c61-4cde-acee-900d239aa617-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:21.026946 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:21.026697 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/4e7a6b98-3c61-4cde-acee-900d239aa617-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:21.026946 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:21.026723 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxs6n\" (UniqueName: \"kubernetes.io/projected/4e7a6b98-3c61-4cde-acee-900d239aa617-kube-api-access-rxs6n\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:21.027298 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:21.027262 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/4e7a6b98-3c61-4cde-acee-900d239aa617-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:21.027491 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:21.027460 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e7a6b98-3c61-4cde-acee-900d239aa617-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:21.027614 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:21.027467 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/4e7a6b98-3c61-4cde-acee-900d239aa617-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:21.027677 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:21.027642 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/4e7a6b98-3c61-4cde-acee-900d239aa617-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:21.029389 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:21.029367 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/4e7a6b98-3c61-4cde-acee-900d239aa617-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:21.029598 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:21.029576 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e7a6b98-3c61-4cde-acee-900d239aa617-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:21.035362 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:21.035340 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/4e7a6b98-3c61-4cde-acee-900d239aa617-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:21.039443 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:21.039414 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxs6n\" (UniqueName: \"kubernetes.io/projected/4e7a6b98-3c61-4cde-acee-900d239aa617-kube-api-access-rxs6n\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:22.299068 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:22.299020 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-ccl7x" event={"ID":"f1630767-8bfc-46fe-b014-d05cef42acae","Type":"ContainerStarted","Data":"d5da9a04d6d2b248ed92c82d89bff2aaea7dfadb66a67dead5def96b02c0e6b1"} Oct 31 00:06:22.299488 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:22.299174 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-ccl7x" Oct 31 00:06:22.319026 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:22.318976 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-6644769df5-ccl7x" podStartSLOduration=1.9267487239999999 podStartE2EDuration="6.318943422s" totalImagesPullingTime="4.392194698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:06:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:06:22.317581882 +0000 UTC m=+418.954400602" watchObservedRunningTime="2026-10-31 00:06:22.318943422 +0000 UTC m=+418.955762138" Oct 31 00:06:24.656811 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:24.656775 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:24.664338 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:24.664318 5274 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:24.757728 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:24.757701 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:24.757891 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:24.757875 5274 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:24.761720 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:24.761703 5274 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") DevicePath \"csi-9ce3c80b5d5deac9dfdb953db4fb017080b654a85db76a2937e67cd541d89057\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:24.938492 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:24.938456 5274 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/151619720cb39d40781bf258206fb0be746b77e5f55879e189e4cf0d6e46dfb0/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:24.951580 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:24.951557 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:25.053898 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:25.053878 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-7d7qm\"" Oct 31 00:06:25.061047 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:25.061020 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:25.343590 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:25.343563 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:06:25.346032 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:06:25.346003 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e7a6b98_3c61_4cde_acee_900d239aa617.slice/crio-eb01244fb3614bf06f7229a4706657f1d4daf751048e0687d159d1b609e9dfdb WatchSource:0}: Error finding container eb01244fb3614bf06f7229a4706657f1d4daf751048e0687d159d1b609e9dfdb: Status 404 returned error can't find the container with id eb01244fb3614bf06f7229a4706657f1d4daf751048e0687d159d1b609e9dfdb Oct 31 00:06:26.318476 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:26.318442 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"4e7a6b98-3c61-4cde-acee-900d239aa617","Type":"ContainerStarted","Data":"eb01244fb3614bf06f7229a4706657f1d4daf751048e0687d159d1b609e9dfdb"} Oct 31 00:06:29.336465 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:29.336424 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"4e7a6b98-3c61-4cde-acee-900d239aa617","Type":"ContainerStarted","Data":"2d364d99c13a5f296f268e7b1745d53bdfce41aed1d63cee75a94207aba5f8e2"} Oct 31 00:06:32.353446 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:32.353411 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"4e7a6b98-3c61-4cde-acee-900d239aa617","Type":"ContainerStarted","Data":"43006d85f0c77e49919a9fa234c17cf6d7fe5ba60a93d3808a981ea9a4eb80ab"} Oct 31 00:06:33.305451 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:33.305423 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-ccl7x" Oct 31 00:06:34.440192 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:34.440151 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-ccl7x"] Oct 31 00:06:34.440684 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:34.440446 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-6644769df5-ccl7x" podUID="f1630767-8bfc-46fe-b014-d05cef42acae" containerName="korrel8r" containerID="cri-o://d5da9a04d6d2b248ed92c82d89bff2aaea7dfadb66a67dead5def96b02c0e6b1" gracePeriod=30 Oct 31 00:06:34.689515 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:34.689482 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:06:34.751767 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:34.751743 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-ccl7x" Oct 31 00:06:34.845581 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:34.845547 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1630767-8bfc-46fe-b014-d05cef42acae-config\" (UniqueName: \"kubernetes.io/configmap/f1630767-8bfc-46fe-b014-d05cef42acae-config\") pod \"f1630767-8bfc-46fe-b014-d05cef42acae\" (UID: \"f1630767-8bfc-46fe-b014-d05cef42acae\") " Oct 31 00:06:34.845674 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:34.845599 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1630767-8bfc-46fe-b014-d05cef42acae-kube-api-access-jbjlw\" (UniqueName: \"kubernetes.io/projected/f1630767-8bfc-46fe-b014-d05cef42acae-kube-api-access-jbjlw\") pod \"f1630767-8bfc-46fe-b014-d05cef42acae\" (UID: \"f1630767-8bfc-46fe-b014-d05cef42acae\") " Oct 31 00:06:34.845718 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:34.845673 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1630767-8bfc-46fe-b014-d05cef42acae-tls\" (UniqueName: \"kubernetes.io/secret/f1630767-8bfc-46fe-b014-d05cef42acae-tls\") pod \"f1630767-8bfc-46fe-b014-d05cef42acae\" (UID: \"f1630767-8bfc-46fe-b014-d05cef42acae\") " Oct 31 00:06:34.845974 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:34.845917 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1630767-8bfc-46fe-b014-d05cef42acae-config" pod "f1630767-8bfc-46fe-b014-d05cef42acae" (UID: "f1630767-8bfc-46fe-b014-d05cef42acae"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:06:34.847776 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:34.847748 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1630767-8bfc-46fe-b014-d05cef42acae-tls" pod "f1630767-8bfc-46fe-b014-d05cef42acae" (UID: "f1630767-8bfc-46fe-b014-d05cef42acae"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:06:34.847856 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:34.847830 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1630767-8bfc-46fe-b014-d05cef42acae-kube-api-access-jbjlw" pod "f1630767-8bfc-46fe-b014-d05cef42acae" (UID: "f1630767-8bfc-46fe-b014-d05cef42acae"). InnerVolumeSpecName "kube-api-access-jbjlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:06:34.947361 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:34.947317 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f1630767-8bfc-46fe-b014-d05cef42acae-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:34.947361 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:34.947356 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f1630767-8bfc-46fe-b014-d05cef42acae-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:34.947361 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:34.947367 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbjlw\" (UniqueName: \"kubernetes.io/projected/f1630767-8bfc-46fe-b014-d05cef42acae-kube-api-access-jbjlw\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:35.369340 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.369302 5274 generic.go:356] "Generic (PLEG): container finished" podID="f1630767-8bfc-46fe-b014-d05cef42acae" containerID="d5da9a04d6d2b248ed92c82d89bff2aaea7dfadb66a67dead5def96b02c0e6b1" exitCode=0 Oct 31 00:06:35.369340 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.369344 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-ccl7x" Oct 31 00:06:35.369653 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.369383 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-ccl7x" event={"ID":"f1630767-8bfc-46fe-b014-d05cef42acae","Type":"ContainerDied","Data":"d5da9a04d6d2b248ed92c82d89bff2aaea7dfadb66a67dead5def96b02c0e6b1"} Oct 31 00:06:35.369653 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.369438 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-ccl7x" event={"ID":"f1630767-8bfc-46fe-b014-d05cef42acae","Type":"ContainerDied","Data":"bf4b0774bf942fc692e5aef92b83d892d06c74b7ebbaf68d94f039887514c1cf"} Oct 31 00:06:35.369653 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.369468 5274 scope.go:122] "RemoveContainer" containerID="d5da9a04d6d2b248ed92c82d89bff2aaea7dfadb66a67dead5def96b02c0e6b1" Oct 31 00:06:35.371969 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.371758 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"4e7a6b98-3c61-4cde-acee-900d239aa617","Type":"ContainerStarted","Data":"64c26f36d20844a6013cc3f3f7c0d479c977d7fcc0af3e440fbc8906924dff4d"} Oct 31 00:06:35.371969 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.371787 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="4e7a6b98-3c61-4cde-acee-900d239aa617" containerName="tempo" containerID="cri-o://2d364d99c13a5f296f268e7b1745d53bdfce41aed1d63cee75a94207aba5f8e2" gracePeriod=30 Oct 31 00:06:35.371969 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.371819 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="4e7a6b98-3c61-4cde-acee-900d239aa617" containerName="tempo-gateway" containerID="cri-o://43006d85f0c77e49919a9fa234c17cf6d7fe5ba60a93d3808a981ea9a4eb80ab" gracePeriod=30 Oct 31 00:06:35.372167 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.371930 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="4e7a6b98-3c61-4cde-acee-900d239aa617" containerName="tempo-gateway-opa" containerID="cri-o://64c26f36d20844a6013cc3f3f7c0d479c977d7fcc0af3e440fbc8906924dff4d" gracePeriod=30 Oct 31 00:06:35.372167 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.371822 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:35.372377 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.372184 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:35.372377 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.372209 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:35.375092 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.375045 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:35.378192 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.378165 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="4e7a6b98-3c61-4cde-acee-900d239aa617" containerName="tempo-gateway" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:06:35.383629 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.383611 5274 scope.go:122] "RemoveContainer" containerID="d5da9a04d6d2b248ed92c82d89bff2aaea7dfadb66a67dead5def96b02c0e6b1" Oct 31 00:06:35.383948 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:06:35.383928 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5da9a04d6d2b248ed92c82d89bff2aaea7dfadb66a67dead5def96b02c0e6b1\": container with ID starting with d5da9a04d6d2b248ed92c82d89bff2aaea7dfadb66a67dead5def96b02c0e6b1 not found: ID does not exist" containerID="d5da9a04d6d2b248ed92c82d89bff2aaea7dfadb66a67dead5def96b02c0e6b1" Oct 31 00:06:35.384030 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.383975 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5da9a04d6d2b248ed92c82d89bff2aaea7dfadb66a67dead5def96b02c0e6b1"} err="failed to get container status \"d5da9a04d6d2b248ed92c82d89bff2aaea7dfadb66a67dead5def96b02c0e6b1\": rpc error: code = NotFound desc = could not find container \"d5da9a04d6d2b248ed92c82d89bff2aaea7dfadb66a67dead5def96b02c0e6b1\": container with ID starting with d5da9a04d6d2b248ed92c82d89bff2aaea7dfadb66a67dead5def96b02c0e6b1 not found: ID does not exist" Oct 31 00:06:35.400403 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.400366 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=9.056680586 podStartE2EDuration="18.4003538s" totalImagesPullingTime="9.343673214s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:06:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:06:35.398816011 +0000 UTC m=+432.035634744" watchObservedRunningTime="2026-10-31 00:06:35.4003538 +0000 UTC m=+432.037172570" Oct 31 00:06:35.438809 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.438787 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-ccl7x"] Oct 31 00:06:35.448977 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.446909 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-ccl7x"] Oct 31 00:06:35.480473 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.480447 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1630767-8bfc-46fe-b014-d05cef42acae" path="/var/lib/kubelet/pods/f1630767-8bfc-46fe-b014-d05cef42acae/volumes" Oct 31 00:06:35.713618 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.713597 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:35.756186 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.756155 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e7a6b98-3c61-4cde-acee-900d239aa617-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/4e7a6b98-3c61-4cde-acee-900d239aa617-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"4e7a6b98-3c61-4cde-acee-900d239aa617\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") " Oct 31 00:06:35.756332 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.756201 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e7a6b98-3c61-4cde-acee-900d239aa617-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/4e7a6b98-3c61-4cde-acee-900d239aa617-gateway-rbac\") pod \"4e7a6b98-3c61-4cde-acee-900d239aa617\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") " Oct 31 00:06:35.756332 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.756237 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e7a6b98-3c61-4cde-acee-900d239aa617-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/4e7a6b98-3c61-4cde-acee-900d239aa617-gateway-tenants\") pod \"4e7a6b98-3c61-4cde-acee-900d239aa617\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") " Oct 31 00:06:35.756448 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.756342 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"4e7a6b98-3c61-4cde-acee-900d239aa617\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") " Oct 31 00:06:35.756585 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.756557 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e7a6b98-3c61-4cde-acee-900d239aa617-tempo-data-science-tempomonolithic-serving-cabundle" pod "4e7a6b98-3c61-4cde-acee-900d239aa617" (UID: "4e7a6b98-3c61-4cde-acee-900d239aa617"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:06:35.756642 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.756586 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e7a6b98-3c61-4cde-acee-900d239aa617-gateway-rbac" pod "4e7a6b98-3c61-4cde-acee-900d239aa617" (UID: "4e7a6b98-3c61-4cde-acee-900d239aa617"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:06:35.756642 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.756617 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e7a6b98-3c61-4cde-acee-900d239aa617-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/4e7a6b98-3c61-4cde-acee-900d239aa617-tempo-data-science-tempomonolithic-mtls\") pod \"4e7a6b98-3c61-4cde-acee-900d239aa617\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") " Oct 31 00:06:35.756739 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.756674 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e7a6b98-3c61-4cde-acee-900d239aa617-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e7a6b98-3c61-4cde-acee-900d239aa617-tempo-data-science-tempomonolithic-ca-bundle\") pod \"4e7a6b98-3c61-4cde-acee-900d239aa617\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") " Oct 31 00:06:35.756853 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.756833 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e7a6b98-3c61-4cde-acee-900d239aa617-kube-api-access-rxs6n\" (UniqueName: \"kubernetes.io/projected/4e7a6b98-3c61-4cde-acee-900d239aa617-kube-api-access-rxs6n\") pod \"4e7a6b98-3c61-4cde-acee-900d239aa617\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") " Oct 31 00:06:35.757130 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.757090 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e7a6b98-3c61-4cde-acee-900d239aa617-tempo-data-science-tempomonolithic-ca-bundle" pod "4e7a6b98-3c61-4cde-acee-900d239aa617" (UID: "4e7a6b98-3c61-4cde-acee-900d239aa617"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:06:35.757259 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.757225 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e7a6b98-3c61-4cde-acee-900d239aa617-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e7a6b98-3c61-4cde-acee-900d239aa617-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"4e7a6b98-3c61-4cde-acee-900d239aa617\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") " Oct 31 00:06:35.757328 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.757282 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e7a6b98-3c61-4cde-acee-900d239aa617-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/4e7a6b98-3c61-4cde-acee-900d239aa617-tempo-conf\") pod \"4e7a6b98-3c61-4cde-acee-900d239aa617\" (UID: \"4e7a6b98-3c61-4cde-acee-900d239aa617\") " Oct 31 00:06:35.757652 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.757620 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e7a6b98-3c61-4cde-acee-900d239aa617-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:35.757652 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.757649 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/4e7a6b98-3c61-4cde-acee-900d239aa617-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:35.757813 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.757667 5274 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/4e7a6b98-3c61-4cde-acee-900d239aa617-gateway-rbac\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:35.757813 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.757751 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e7a6b98-3c61-4cde-acee-900d239aa617-tempo-conf" pod "4e7a6b98-3c61-4cde-acee-900d239aa617" (UID: "4e7a6b98-3c61-4cde-acee-900d239aa617"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:06:35.758980 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.758916 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e7a6b98-3c61-4cde-acee-900d239aa617-gateway-tenants" pod "4e7a6b98-3c61-4cde-acee-900d239aa617" (UID: "4e7a6b98-3c61-4cde-acee-900d239aa617"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:06:35.760414 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.759346 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e7a6b98-3c61-4cde-acee-900d239aa617-tempo-data-science-tempomonolithic-mtls" pod "4e7a6b98-3c61-4cde-acee-900d239aa617" (UID: "4e7a6b98-3c61-4cde-acee-900d239aa617"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:06:35.760809 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.760682 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e7a6b98-3c61-4cde-acee-900d239aa617-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "4e7a6b98-3c61-4cde-acee-900d239aa617" (UID: "4e7a6b98-3c61-4cde-acee-900d239aa617"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:06:35.760915 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.760863 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e7a6b98-3c61-4cde-acee-900d239aa617-kube-api-access-rxs6n" pod "4e7a6b98-3c61-4cde-acee-900d239aa617" (UID: "4e7a6b98-3c61-4cde-acee-900d239aa617"). InnerVolumeSpecName "kube-api-access-rxs6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:06:35.761122 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.761105 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062" pod "4e7a6b98-3c61-4cde-acee-900d239aa617" (UID: "4e7a6b98-3c61-4cde-acee-900d239aa617"). InnerVolumeSpecName "pvc-4491ac70-4597-43a7-a792-f0813314e433". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:06:35.858386 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.858365 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxs6n\" (UniqueName: \"kubernetes.io/projected/4e7a6b98-3c61-4cde-acee-900d239aa617-kube-api-access-rxs6n\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:35.858386 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.858386 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e7a6b98-3c61-4cde-acee-900d239aa617-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:35.858590 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.858397 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/4e7a6b98-3c61-4cde-acee-900d239aa617-tempo-conf\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:35.858590 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.858406 5274 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/4e7a6b98-3c61-4cde-acee-900d239aa617-gateway-tenants\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:35.858590 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.858433 5274 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" " Oct 31 00:06:35.858590 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.858443 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/4e7a6b98-3c61-4cde-acee-900d239aa617-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:35.868559 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.868543 5274 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-4491ac70-4597-43a7-a792-f0813314e433" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062") on node "ip-10-0-129-70.us-east-2.compute.internal" Oct 31 00:06:35.959309 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:35.959246 5274 reconciler_common.go:299] "Volume detached for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"csi-9ce3c80b5d5deac9dfdb953db4fb017080b654a85db76a2937e67cd541d89057\"" Oct 31 00:06:36.379365 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:36.379336 5274 generic.go:356] "Generic (PLEG): container finished" podID="4e7a6b98-3c61-4cde-acee-900d239aa617" containerID="64c26f36d20844a6013cc3f3f7c0d479c977d7fcc0af3e440fbc8906924dff4d" exitCode=0 Oct 31 00:06:36.379365 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:36.379360 5274 generic.go:356] "Generic (PLEG): container finished" podID="4e7a6b98-3c61-4cde-acee-900d239aa617" containerID="43006d85f0c77e49919a9fa234c17cf6d7fe5ba60a93d3808a981ea9a4eb80ab" exitCode=0 Oct 31 00:06:36.379365 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:36.379368 5274 generic.go:356] "Generic (PLEG): container finished" podID="4e7a6b98-3c61-4cde-acee-900d239aa617" containerID="2d364d99c13a5f296f268e7b1745d53bdfce41aed1d63cee75a94207aba5f8e2" exitCode=0 Oct 31 00:06:36.379606 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:36.379391 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:06:36.379606 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:36.379409 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"4e7a6b98-3c61-4cde-acee-900d239aa617","Type":"ContainerDied","Data":"64c26f36d20844a6013cc3f3f7c0d479c977d7fcc0af3e440fbc8906924dff4d"} Oct 31 00:06:36.379606 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:36.379451 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"4e7a6b98-3c61-4cde-acee-900d239aa617","Type":"ContainerDied","Data":"43006d85f0c77e49919a9fa234c17cf6d7fe5ba60a93d3808a981ea9a4eb80ab"} Oct 31 00:06:36.379606 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:36.379469 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"4e7a6b98-3c61-4cde-acee-900d239aa617","Type":"ContainerDied","Data":"2d364d99c13a5f296f268e7b1745d53bdfce41aed1d63cee75a94207aba5f8e2"} Oct 31 00:06:36.379606 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:36.379483 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"4e7a6b98-3c61-4cde-acee-900d239aa617","Type":"ContainerDied","Data":"eb01244fb3614bf06f7229a4706657f1d4daf751048e0687d159d1b609e9dfdb"} Oct 31 00:06:36.379606 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:36.379502 5274 scope.go:122] "RemoveContainer" containerID="64c26f36d20844a6013cc3f3f7c0d479c977d7fcc0af3e440fbc8906924dff4d" Oct 31 00:06:36.388220 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:36.388204 5274 scope.go:122] "RemoveContainer" containerID="43006d85f0c77e49919a9fa234c17cf6d7fe5ba60a93d3808a981ea9a4eb80ab" Oct 31 00:06:36.395655 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:36.395640 5274 scope.go:122] "RemoveContainer" containerID="2d364d99c13a5f296f268e7b1745d53bdfce41aed1d63cee75a94207aba5f8e2" Oct 31 00:06:36.403213 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:36.403192 5274 scope.go:122] "RemoveContainer" containerID="64c26f36d20844a6013cc3f3f7c0d479c977d7fcc0af3e440fbc8906924dff4d" Oct 31 00:06:36.403599 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:06:36.403554 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64c26f36d20844a6013cc3f3f7c0d479c977d7fcc0af3e440fbc8906924dff4d\": container with ID starting with 64c26f36d20844a6013cc3f3f7c0d479c977d7fcc0af3e440fbc8906924dff4d not found: ID does not exist" containerID="64c26f36d20844a6013cc3f3f7c0d479c977d7fcc0af3e440fbc8906924dff4d" Oct 31 00:06:36.403691 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:36.403611 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64c26f36d20844a6013cc3f3f7c0d479c977d7fcc0af3e440fbc8906924dff4d"} err="failed to get container status \"64c26f36d20844a6013cc3f3f7c0d479c977d7fcc0af3e440fbc8906924dff4d\": rpc error: code = NotFound desc = could not find container \"64c26f36d20844a6013cc3f3f7c0d479c977d7fcc0af3e440fbc8906924dff4d\": container with ID starting with 64c26f36d20844a6013cc3f3f7c0d479c977d7fcc0af3e440fbc8906924dff4d not found: ID does not exist" Oct 31 00:06:36.403691 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:36.403634 5274 scope.go:122] "RemoveContainer" containerID="43006d85f0c77e49919a9fa234c17cf6d7fe5ba60a93d3808a981ea9a4eb80ab" Oct 31 00:06:36.403945 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:06:36.403919 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43006d85f0c77e49919a9fa234c17cf6d7fe5ba60a93d3808a981ea9a4eb80ab\": container with ID starting with 43006d85f0c77e49919a9fa234c17cf6d7fe5ba60a93d3808a981ea9a4eb80ab not found: ID does not exist" containerID="43006d85f0c77e49919a9fa234c17cf6d7fe5ba60a93d3808a981ea9a4eb80ab" Oct 31 00:06:36.404016 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:36.403968 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43006d85f0c77e49919a9fa234c17cf6d7fe5ba60a93d3808a981ea9a4eb80ab"} err="failed to get container status \"43006d85f0c77e49919a9fa234c17cf6d7fe5ba60a93d3808a981ea9a4eb80ab\": rpc error: code = NotFound desc = could not find container \"43006d85f0c77e49919a9fa234c17cf6d7fe5ba60a93d3808a981ea9a4eb80ab\": container with ID starting with 43006d85f0c77e49919a9fa234c17cf6d7fe5ba60a93d3808a981ea9a4eb80ab not found: ID does not exist" Oct 31 00:06:36.404016 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:36.403989 5274 scope.go:122] "RemoveContainer" containerID="2d364d99c13a5f296f268e7b1745d53bdfce41aed1d63cee75a94207aba5f8e2" Oct 31 00:06:36.404247 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:06:36.404229 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d364d99c13a5f296f268e7b1745d53bdfce41aed1d63cee75a94207aba5f8e2\": container with ID starting with 2d364d99c13a5f296f268e7b1745d53bdfce41aed1d63cee75a94207aba5f8e2 not found: ID does not exist" containerID="2d364d99c13a5f296f268e7b1745d53bdfce41aed1d63cee75a94207aba5f8e2" Oct 31 00:06:36.404297 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:36.404254 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d364d99c13a5f296f268e7b1745d53bdfce41aed1d63cee75a94207aba5f8e2"} err="failed to get container status \"2d364d99c13a5f296f268e7b1745d53bdfce41aed1d63cee75a94207aba5f8e2\": rpc error: code = NotFound desc = could not find container \"2d364d99c13a5f296f268e7b1745d53bdfce41aed1d63cee75a94207aba5f8e2\": container with ID starting with 2d364d99c13a5f296f268e7b1745d53bdfce41aed1d63cee75a94207aba5f8e2 not found: ID does not exist" Oct 31 00:06:36.404297 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:36.404269 5274 scope.go:122] "RemoveContainer" containerID="64c26f36d20844a6013cc3f3f7c0d479c977d7fcc0af3e440fbc8906924dff4d" Oct 31 00:06:36.404556 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:36.404527 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64c26f36d20844a6013cc3f3f7c0d479c977d7fcc0af3e440fbc8906924dff4d"} err="failed to get container status \"64c26f36d20844a6013cc3f3f7c0d479c977d7fcc0af3e440fbc8906924dff4d\": rpc error: code = NotFound desc = could not find container \"64c26f36d20844a6013cc3f3f7c0d479c977d7fcc0af3e440fbc8906924dff4d\": container with ID starting with 64c26f36d20844a6013cc3f3f7c0d479c977d7fcc0af3e440fbc8906924dff4d not found: ID does not exist" Oct 31 00:06:36.404556 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:36.404555 5274 scope.go:122] "RemoveContainer" containerID="43006d85f0c77e49919a9fa234c17cf6d7fe5ba60a93d3808a981ea9a4eb80ab" Oct 31 00:06:36.404802 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:36.404787 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43006d85f0c77e49919a9fa234c17cf6d7fe5ba60a93d3808a981ea9a4eb80ab"} err="failed to get container status \"43006d85f0c77e49919a9fa234c17cf6d7fe5ba60a93d3808a981ea9a4eb80ab\": rpc error: code = NotFound desc = could not find container \"43006d85f0c77e49919a9fa234c17cf6d7fe5ba60a93d3808a981ea9a4eb80ab\": container with ID starting with 43006d85f0c77e49919a9fa234c17cf6d7fe5ba60a93d3808a981ea9a4eb80ab not found: ID does not exist" Oct 31 00:06:36.404850 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:36.404804 5274 scope.go:122] "RemoveContainer" containerID="2d364d99c13a5f296f268e7b1745d53bdfce41aed1d63cee75a94207aba5f8e2" Oct 31 00:06:36.405064 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:36.405047 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d364d99c13a5f296f268e7b1745d53bdfce41aed1d63cee75a94207aba5f8e2"} err="failed to get container status \"2d364d99c13a5f296f268e7b1745d53bdfce41aed1d63cee75a94207aba5f8e2\": rpc error: code = NotFound desc = could not find container \"2d364d99c13a5f296f268e7b1745d53bdfce41aed1d63cee75a94207aba5f8e2\": container with ID starting with 2d364d99c13a5f296f268e7b1745d53bdfce41aed1d63cee75a94207aba5f8e2 not found: ID does not exist" Oct 31 00:06:36.405113 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:36.405064 5274 scope.go:122] "RemoveContainer" containerID="64c26f36d20844a6013cc3f3f7c0d479c977d7fcc0af3e440fbc8906924dff4d" Oct 31 00:06:36.405153 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:36.405118 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:06:36.405292 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:36.405275 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64c26f36d20844a6013cc3f3f7c0d479c977d7fcc0af3e440fbc8906924dff4d"} err="failed to get container status \"64c26f36d20844a6013cc3f3f7c0d479c977d7fcc0af3e440fbc8906924dff4d\": rpc error: code = NotFound desc = could not find container \"64c26f36d20844a6013cc3f3f7c0d479c977d7fcc0af3e440fbc8906924dff4d\": container with ID starting with 64c26f36d20844a6013cc3f3f7c0d479c977d7fcc0af3e440fbc8906924dff4d not found: ID does not exist" Oct 31 00:06:36.405333 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:36.405293 5274 scope.go:122] "RemoveContainer" containerID="43006d85f0c77e49919a9fa234c17cf6d7fe5ba60a93d3808a981ea9a4eb80ab" Oct 31 00:06:36.405497 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:36.405475 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43006d85f0c77e49919a9fa234c17cf6d7fe5ba60a93d3808a981ea9a4eb80ab"} err="failed to get container status \"43006d85f0c77e49919a9fa234c17cf6d7fe5ba60a93d3808a981ea9a4eb80ab\": rpc error: code = NotFound desc = could not find container \"43006d85f0c77e49919a9fa234c17cf6d7fe5ba60a93d3808a981ea9a4eb80ab\": container with ID starting with 43006d85f0c77e49919a9fa234c17cf6d7fe5ba60a93d3808a981ea9a4eb80ab not found: ID does not exist" Oct 31 00:06:36.405497 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:36.405491 5274 scope.go:122] "RemoveContainer" containerID="2d364d99c13a5f296f268e7b1745d53bdfce41aed1d63cee75a94207aba5f8e2" Oct 31 00:06:36.406512 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:36.406491 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d364d99c13a5f296f268e7b1745d53bdfce41aed1d63cee75a94207aba5f8e2"} err="failed to get container status \"2d364d99c13a5f296f268e7b1745d53bdfce41aed1d63cee75a94207aba5f8e2\": rpc error: code = NotFound desc = could not find container \"2d364d99c13a5f296f268e7b1745d53bdfce41aed1d63cee75a94207aba5f8e2\": container with ID starting with 2d364d99c13a5f296f268e7b1745d53bdfce41aed1d63cee75a94207aba5f8e2 not found: ID does not exist" Oct 31 00:06:36.408073 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:36.408056 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:06:37.480433 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:37.480396 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e7a6b98-3c61-4cde-acee-900d239aa617" path="/var/lib/kubelet/pods/4e7a6b98-3c61-4cde-acee-900d239aa617/volumes" Oct 31 00:06:38.183033 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:38.183005 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-84f5b7859-cvxgp"] Oct 31 00:06:38.183333 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:38.183322 5274 cpu_manager.go:414] "containerMap: removing container" podUID="4e7a6b98-3c61-4cde-acee-900d239aa617" containerName="tempo" Oct 31 00:06:38.183380 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:38.183335 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e7a6b98-3c61-4cde-acee-900d239aa617" containerName="tempo" Oct 31 00:06:38.183380 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:38.183352 5274 cpu_manager.go:414] "containerMap: removing container" podUID="4e7a6b98-3c61-4cde-acee-900d239aa617" containerName="tempo-gateway" Oct 31 00:06:38.183380 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:38.183358 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e7a6b98-3c61-4cde-acee-900d239aa617" containerName="tempo-gateway" Oct 31 00:06:38.183380 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:38.183375 5274 cpu_manager.go:414] "containerMap: removing container" podUID="f1630767-8bfc-46fe-b014-d05cef42acae" containerName="korrel8r" Oct 31 00:06:38.183380 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:38.183380 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1630767-8bfc-46fe-b014-d05cef42acae" containerName="korrel8r" Oct 31 00:06:38.183531 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:38.183387 5274 cpu_manager.go:414] "containerMap: removing container" podUID="4e7a6b98-3c61-4cde-acee-900d239aa617" containerName="tempo-gateway-opa" Oct 31 00:06:38.183531 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:38.183392 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e7a6b98-3c61-4cde-acee-900d239aa617" containerName="tempo-gateway-opa" Oct 31 00:06:38.183531 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:38.183453 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e7a6b98-3c61-4cde-acee-900d239aa617" containerName="tempo" Oct 31 00:06:38.183531 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:38.183463 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e7a6b98-3c61-4cde-acee-900d239aa617" containerName="tempo-gateway" Oct 31 00:06:38.183531 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:38.183471 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1630767-8bfc-46fe-b014-d05cef42acae" containerName="korrel8r" Oct 31 00:06:38.183531 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:38.183476 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e7a6b98-3c61-4cde-acee-900d239aa617" containerName="tempo-gateway-opa" Oct 31 00:06:38.186505 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:38.186488 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-84f5b7859-cvxgp" Oct 31 00:06:38.192222 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:38.192201 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 31 00:06:38.196176 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:38.196045 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:06:38.196176 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:38.196093 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 31 00:06:38.196176 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:38.196151 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:06:38.197940 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:38.197913 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-84f5b7859-cvxgp"] Oct 31 00:06:38.258967 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:38.258932 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-4zw8t\"" Oct 31 00:06:38.274922 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:38.274901 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/07b0112b-ba5f-4356-8efc-256f5c764570-config\") pod \"korrel8r-84f5b7859-cvxgp\" (UID: \"07b0112b-ba5f-4356-8efc-256f5c764570\") " pod="redhat-ods-monitoring/korrel8r-84f5b7859-cvxgp" Oct 31 00:06:38.275040 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:38.274929 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/07b0112b-ba5f-4356-8efc-256f5c764570-tls\") pod \"korrel8r-84f5b7859-cvxgp\" (UID: \"07b0112b-ba5f-4356-8efc-256f5c764570\") " pod="redhat-ods-monitoring/korrel8r-84f5b7859-cvxgp" Oct 31 00:06:38.275087 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:38.275070 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-449xl\" (UniqueName: \"kubernetes.io/projected/07b0112b-ba5f-4356-8efc-256f5c764570-kube-api-access-449xl\") pod \"korrel8r-84f5b7859-cvxgp\" (UID: \"07b0112b-ba5f-4356-8efc-256f5c764570\") " pod="redhat-ods-monitoring/korrel8r-84f5b7859-cvxgp" Oct 31 00:06:38.375394 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:38.375361 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-449xl\" (UniqueName: \"kubernetes.io/projected/07b0112b-ba5f-4356-8efc-256f5c764570-kube-api-access-449xl\") pod \"korrel8r-84f5b7859-cvxgp\" (UID: \"07b0112b-ba5f-4356-8efc-256f5c764570\") " pod="redhat-ods-monitoring/korrel8r-84f5b7859-cvxgp" Oct 31 00:06:38.375514 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:38.375412 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/07b0112b-ba5f-4356-8efc-256f5c764570-config\") pod \"korrel8r-84f5b7859-cvxgp\" (UID: \"07b0112b-ba5f-4356-8efc-256f5c764570\") " pod="redhat-ods-monitoring/korrel8r-84f5b7859-cvxgp" Oct 31 00:06:38.375514 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:38.375432 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/07b0112b-ba5f-4356-8efc-256f5c764570-tls\") pod \"korrel8r-84f5b7859-cvxgp\" (UID: \"07b0112b-ba5f-4356-8efc-256f5c764570\") " pod="redhat-ods-monitoring/korrel8r-84f5b7859-cvxgp" Oct 31 00:06:38.376065 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:38.376045 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/07b0112b-ba5f-4356-8efc-256f5c764570-config\") pod \"korrel8r-84f5b7859-cvxgp\" (UID: \"07b0112b-ba5f-4356-8efc-256f5c764570\") " pod="redhat-ods-monitoring/korrel8r-84f5b7859-cvxgp" Oct 31 00:06:38.377898 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:38.377877 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/07b0112b-ba5f-4356-8efc-256f5c764570-tls\") pod \"korrel8r-84f5b7859-cvxgp\" (UID: \"07b0112b-ba5f-4356-8efc-256f5c764570\") " pod="redhat-ods-monitoring/korrel8r-84f5b7859-cvxgp" Oct 31 00:06:38.387439 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:38.387414 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-449xl\" (UniqueName: \"kubernetes.io/projected/07b0112b-ba5f-4356-8efc-256f5c764570-kube-api-access-449xl\") pod \"korrel8r-84f5b7859-cvxgp\" (UID: \"07b0112b-ba5f-4356-8efc-256f5c764570\") " pod="redhat-ods-monitoring/korrel8r-84f5b7859-cvxgp" Oct 31 00:06:38.499372 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:38.499297 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-84f5b7859-cvxgp" Oct 31 00:06:38.780141 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:38.780111 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-84f5b7859-cvxgp"] Oct 31 00:06:38.781436 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:06:38.781412 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b0112b_ba5f_4356_8efc_256f5c764570.slice/crio-dd98b915229a5534f7b4c63d6957409bdf6ced9bfae8440cece5539c476cbdbd WatchSource:0}: Error finding container dd98b915229a5534f7b4c63d6957409bdf6ced9bfae8440cece5539c476cbdbd: Status 404 returned error can't find the container with id dd98b915229a5534f7b4c63d6957409bdf6ced9bfae8440cece5539c476cbdbd Oct 31 00:06:39.395565 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:39.395508 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-84f5b7859-cvxgp" event={"ID":"07b0112b-ba5f-4356-8efc-256f5c764570","Type":"ContainerStarted","Data":"2c148d100337a9dcc4fb5f865bd50a2b475f548a8f688753b8d9aa1f985aedcc"} Oct 31 00:06:39.395733 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:39.395590 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-84f5b7859-cvxgp" event={"ID":"07b0112b-ba5f-4356-8efc-256f5c764570","Type":"ContainerStarted","Data":"dd98b915229a5534f7b4c63d6957409bdf6ced9bfae8440cece5539c476cbdbd"} Oct 31 00:06:39.395733 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:39.395705 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-84f5b7859-cvxgp" Oct 31 00:06:39.419147 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:39.419102 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-84f5b7859-cvxgp" podStartSLOduration=1.419088995 podStartE2EDuration="1.419088995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:06:39.418058319 +0000 UTC m=+436.054877035" watchObservedRunningTime="2026-10-31 00:06:39.419088995 +0000 UTC m=+436.055907711" Oct 31 00:06:50.403305 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:50.403274 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-84f5b7859-cvxgp" Oct 31 00:06:50.899071 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:06:50.899039 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:06:51.840199 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:51.840165 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-84f5b7859-cvxgp"] Oct 31 00:06:51.840607 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:51.840348 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-84f5b7859-cvxgp" podUID="07b0112b-ba5f-4356-8efc-256f5c764570" containerName="korrel8r" containerID="cri-o://2c148d100337a9dcc4fb5f865bd50a2b475f548a8f688753b8d9aa1f985aedcc" gracePeriod=30 Oct 31 00:06:52.106583 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:52.106530 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-84f5b7859-cvxgp" Oct 31 00:06:52.182785 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:52.182748 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07b0112b-ba5f-4356-8efc-256f5c764570-kube-api-access-449xl\" (UniqueName: \"kubernetes.io/projected/07b0112b-ba5f-4356-8efc-256f5c764570-kube-api-access-449xl\") pod \"07b0112b-ba5f-4356-8efc-256f5c764570\" (UID: \"07b0112b-ba5f-4356-8efc-256f5c764570\") " Oct 31 00:06:52.182986 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:52.182832 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07b0112b-ba5f-4356-8efc-256f5c764570-tls\" (UniqueName: \"kubernetes.io/secret/07b0112b-ba5f-4356-8efc-256f5c764570-tls\") pod \"07b0112b-ba5f-4356-8efc-256f5c764570\" (UID: \"07b0112b-ba5f-4356-8efc-256f5c764570\") " Oct 31 00:06:52.182986 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:52.182942 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07b0112b-ba5f-4356-8efc-256f5c764570-config\" (UniqueName: \"kubernetes.io/configmap/07b0112b-ba5f-4356-8efc-256f5c764570-config\") pod \"07b0112b-ba5f-4356-8efc-256f5c764570\" (UID: \"07b0112b-ba5f-4356-8efc-256f5c764570\") " Oct 31 00:06:52.183338 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:52.183301 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07b0112b-ba5f-4356-8efc-256f5c764570-config" pod "07b0112b-ba5f-4356-8efc-256f5c764570" (UID: "07b0112b-ba5f-4356-8efc-256f5c764570"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:06:52.184853 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:52.184823 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07b0112b-ba5f-4356-8efc-256f5c764570-kube-api-access-449xl" pod "07b0112b-ba5f-4356-8efc-256f5c764570" (UID: "07b0112b-ba5f-4356-8efc-256f5c764570"). InnerVolumeSpecName "kube-api-access-449xl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:06:52.185132 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:52.185101 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07b0112b-ba5f-4356-8efc-256f5c764570-tls" pod "07b0112b-ba5f-4356-8efc-256f5c764570" (UID: "07b0112b-ba5f-4356-8efc-256f5c764570"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:06:52.284192 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:52.284159 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/07b0112b-ba5f-4356-8efc-256f5c764570-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:52.284192 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:52.284186 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-449xl\" (UniqueName: \"kubernetes.io/projected/07b0112b-ba5f-4356-8efc-256f5c764570-kube-api-access-449xl\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:52.284192 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:52.284197 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/07b0112b-ba5f-4356-8efc-256f5c764570-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:52.465201 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:52.465164 5274 generic.go:356] "Generic (PLEG): container finished" podID="07b0112b-ba5f-4356-8efc-256f5c764570" containerID="2c148d100337a9dcc4fb5f865bd50a2b475f548a8f688753b8d9aa1f985aedcc" exitCode=0 Oct 31 00:06:52.465395 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:52.465225 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-84f5b7859-cvxgp" event={"ID":"07b0112b-ba5f-4356-8efc-256f5c764570","Type":"ContainerDied","Data":"2c148d100337a9dcc4fb5f865bd50a2b475f548a8f688753b8d9aa1f985aedcc"} Oct 31 00:06:52.465395 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:52.465230 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-84f5b7859-cvxgp" Oct 31 00:06:52.465395 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:52.465289 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-84f5b7859-cvxgp" event={"ID":"07b0112b-ba5f-4356-8efc-256f5c764570","Type":"ContainerDied","Data":"dd98b915229a5534f7b4c63d6957409bdf6ced9bfae8440cece5539c476cbdbd"} Oct 31 00:06:52.465395 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:52.465311 5274 scope.go:122] "RemoveContainer" containerID="2c148d100337a9dcc4fb5f865bd50a2b475f548a8f688753b8d9aa1f985aedcc" Oct 31 00:06:52.474356 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:52.474339 5274 scope.go:122] "RemoveContainer" containerID="2c148d100337a9dcc4fb5f865bd50a2b475f548a8f688753b8d9aa1f985aedcc" Oct 31 00:06:52.474619 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:06:52.474601 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c148d100337a9dcc4fb5f865bd50a2b475f548a8f688753b8d9aa1f985aedcc\": container with ID starting with 2c148d100337a9dcc4fb5f865bd50a2b475f548a8f688753b8d9aa1f985aedcc not found: ID does not exist" containerID="2c148d100337a9dcc4fb5f865bd50a2b475f548a8f688753b8d9aa1f985aedcc" Oct 31 00:06:52.474682 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:52.474637 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c148d100337a9dcc4fb5f865bd50a2b475f548a8f688753b8d9aa1f985aedcc"} err="failed to get container status \"2c148d100337a9dcc4fb5f865bd50a2b475f548a8f688753b8d9aa1f985aedcc\": rpc error: code = NotFound desc = could not find container \"2c148d100337a9dcc4fb5f865bd50a2b475f548a8f688753b8d9aa1f985aedcc\": container with ID starting with 2c148d100337a9dcc4fb5f865bd50a2b475f548a8f688753b8d9aa1f985aedcc not found: ID does not exist" Oct 31 00:06:52.487110 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:52.487088 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-84f5b7859-cvxgp"] Oct 31 00:06:52.493871 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:52.493851 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-84f5b7859-cvxgp"] Oct 31 00:06:53.480071 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:53.480039 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07b0112b-ba5f-4356-8efc-256f5c764570" path="/var/lib/kubelet/pods/07b0112b-ba5f-4356-8efc-256f5c764570/volumes" Oct 31 00:06:54.277531 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:54.277501 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-58bbdb5dcb-j2zsg"] Oct 31 00:06:54.277844 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:54.277832 5274 cpu_manager.go:414] "containerMap: removing container" podUID="07b0112b-ba5f-4356-8efc-256f5c764570" containerName="korrel8r" Oct 31 00:06:54.277898 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:54.277846 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07b0112b-ba5f-4356-8efc-256f5c764570" containerName="korrel8r" Oct 31 00:06:54.277940 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:54.277917 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="07b0112b-ba5f-4356-8efc-256f5c764570" containerName="korrel8r" Oct 31 00:06:54.284179 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:54.284162 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-j2zsg" Oct 31 00:06:54.290213 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:54.290104 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 31 00:06:54.290213 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:54.290186 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:06:54.290793 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:54.290770 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 31 00:06:54.290929 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:54.290899 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-r5867\"" Oct 31 00:06:54.291293 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:54.291002 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:06:54.291293 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:54.291266 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-58bbdb5dcb-j2zsg"] Oct 31 00:06:54.400477 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:54.400445 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6538808b-2145-4538-9620-6fc37cd63f95-config\") pod \"korrel8r-58bbdb5dcb-j2zsg\" (UID: \"6538808b-2145-4538-9620-6fc37cd63f95\") " pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-j2zsg" Oct 31 00:06:54.400664 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:54.400496 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6538808b-2145-4538-9620-6fc37cd63f95-tls\") pod \"korrel8r-58bbdb5dcb-j2zsg\" (UID: \"6538808b-2145-4538-9620-6fc37cd63f95\") " pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-j2zsg" Oct 31 00:06:54.400664 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:54.400583 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzc29\" (UniqueName: \"kubernetes.io/projected/6538808b-2145-4538-9620-6fc37cd63f95-kube-api-access-pzc29\") pod \"korrel8r-58bbdb5dcb-j2zsg\" (UID: \"6538808b-2145-4538-9620-6fc37cd63f95\") " pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-j2zsg" Oct 31 00:06:54.502111 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:54.502074 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6538808b-2145-4538-9620-6fc37cd63f95-config\") pod \"korrel8r-58bbdb5dcb-j2zsg\" (UID: \"6538808b-2145-4538-9620-6fc37cd63f95\") " pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-j2zsg" Oct 31 00:06:54.502578 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:54.502162 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6538808b-2145-4538-9620-6fc37cd63f95-tls\") pod \"korrel8r-58bbdb5dcb-j2zsg\" (UID: \"6538808b-2145-4538-9620-6fc37cd63f95\") " pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-j2zsg" Oct 31 00:06:54.502578 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:54.502232 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzc29\" (UniqueName: \"kubernetes.io/projected/6538808b-2145-4538-9620-6fc37cd63f95-kube-api-access-pzc29\") pod \"korrel8r-58bbdb5dcb-j2zsg\" (UID: \"6538808b-2145-4538-9620-6fc37cd63f95\") " pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-j2zsg" Oct 31 00:06:54.503303 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:54.503275 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6538808b-2145-4538-9620-6fc37cd63f95-config\") pod \"korrel8r-58bbdb5dcb-j2zsg\" (UID: \"6538808b-2145-4538-9620-6fc37cd63f95\") " pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-j2zsg" Oct 31 00:06:54.506044 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:54.506020 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6538808b-2145-4538-9620-6fc37cd63f95-tls\") pod \"korrel8r-58bbdb5dcb-j2zsg\" (UID: \"6538808b-2145-4538-9620-6fc37cd63f95\") " pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-j2zsg" Oct 31 00:06:54.519490 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:54.519455 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzc29\" (UniqueName: \"kubernetes.io/projected/6538808b-2145-4538-9620-6fc37cd63f95-kube-api-access-pzc29\") pod \"korrel8r-58bbdb5dcb-j2zsg\" (UID: \"6538808b-2145-4538-9620-6fc37cd63f95\") " pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-j2zsg" Oct 31 00:06:54.600125 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:54.600034 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-j2zsg" Oct 31 00:06:54.877227 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:54.877139 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-58bbdb5dcb-j2zsg"] Oct 31 00:06:54.879444 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:06:54.879417 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6538808b_2145_4538_9620_6fc37cd63f95.slice/crio-0b7cf9d60da3342dcfe917a4eed27df517715b86ea079627a2a09d10d9520c27 WatchSource:0}: Error finding container 0b7cf9d60da3342dcfe917a4eed27df517715b86ea079627a2a09d10d9520c27: Status 404 returned error can't find the container with id 0b7cf9d60da3342dcfe917a4eed27df517715b86ea079627a2a09d10d9520c27 Oct 31 00:06:55.482450 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:55.482417 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-j2zsg" event={"ID":"6538808b-2145-4538-9620-6fc37cd63f95","Type":"ContainerStarted","Data":"5cb5af950ad6ca30fca2549a3924bf01809af888088b8e35a189b1fdf9a4f123"} Oct 31 00:06:55.482703 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:55.482467 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-j2zsg" event={"ID":"6538808b-2145-4538-9620-6fc37cd63f95","Type":"ContainerStarted","Data":"0b7cf9d60da3342dcfe917a4eed27df517715b86ea079627a2a09d10d9520c27"} Oct 31 00:06:55.482703 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:55.482518 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-j2zsg" Oct 31 00:06:55.501722 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:06:55.501673 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-j2zsg" podStartSLOduration=1.5016610510000001 podStartE2EDuration="1.501661051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:06:55.500256418 +0000 UTC m=+452.137075130" watchObservedRunningTime="2026-10-31 00:06:55.501661051 +0000 UTC m=+452.138479766" Oct 31 00:07:06.490879 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:06.490804 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-j2zsg" Oct 31 00:07:08.380307 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:08.380274 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-58bbdb5dcb-j2zsg"] Oct 31 00:07:08.380761 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:08.380453 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-j2zsg" podUID="6538808b-2145-4538-9620-6fc37cd63f95" containerName="korrel8r" containerID="cri-o://5cb5af950ad6ca30fca2549a3924bf01809af888088b8e35a189b1fdf9a4f123" gracePeriod=30 Oct 31 00:07:08.548853 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:08.547917 5274 generic.go:356] "Generic (PLEG): container finished" podID="6538808b-2145-4538-9620-6fc37cd63f95" containerID="5cb5af950ad6ca30fca2549a3924bf01809af888088b8e35a189b1fdf9a4f123" exitCode=0 Oct 31 00:07:08.548853 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:08.548159 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-j2zsg" event={"ID":"6538808b-2145-4538-9620-6fc37cd63f95","Type":"ContainerDied","Data":"5cb5af950ad6ca30fca2549a3924bf01809af888088b8e35a189b1fdf9a4f123"} Oct 31 00:07:08.661659 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:08.661636 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-j2zsg" Oct 31 00:07:08.711848 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:08.711819 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6538808b-2145-4538-9620-6fc37cd63f95-tls\" (UniqueName: \"kubernetes.io/secret/6538808b-2145-4538-9620-6fc37cd63f95-tls\") pod \"6538808b-2145-4538-9620-6fc37cd63f95\" (UID: \"6538808b-2145-4538-9620-6fc37cd63f95\") " Oct 31 00:07:08.712013 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:08.711891 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6538808b-2145-4538-9620-6fc37cd63f95-kube-api-access-pzc29\" (UniqueName: \"kubernetes.io/projected/6538808b-2145-4538-9620-6fc37cd63f95-kube-api-access-pzc29\") pod \"6538808b-2145-4538-9620-6fc37cd63f95\" (UID: \"6538808b-2145-4538-9620-6fc37cd63f95\") " Oct 31 00:07:08.712013 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:08.712001 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6538808b-2145-4538-9620-6fc37cd63f95-config\" (UniqueName: \"kubernetes.io/configmap/6538808b-2145-4538-9620-6fc37cd63f95-config\") pod \"6538808b-2145-4538-9620-6fc37cd63f95\" (UID: \"6538808b-2145-4538-9620-6fc37cd63f95\") " Oct 31 00:07:08.712367 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:08.712341 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6538808b-2145-4538-9620-6fc37cd63f95-config" pod "6538808b-2145-4538-9620-6fc37cd63f95" (UID: "6538808b-2145-4538-9620-6fc37cd63f95"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:07:08.713860 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:08.713836 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6538808b-2145-4538-9620-6fc37cd63f95-kube-api-access-pzc29" pod "6538808b-2145-4538-9620-6fc37cd63f95" (UID: "6538808b-2145-4538-9620-6fc37cd63f95"). InnerVolumeSpecName "kube-api-access-pzc29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:07:08.713913 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:08.713880 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6538808b-2145-4538-9620-6fc37cd63f95-tls" pod "6538808b-2145-4538-9620-6fc37cd63f95" (UID: "6538808b-2145-4538-9620-6fc37cd63f95"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:07:08.813564 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:08.813534 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6538808b-2145-4538-9620-6fc37cd63f95-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:08.813564 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:08.813561 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzc29\" (UniqueName: \"kubernetes.io/projected/6538808b-2145-4538-9620-6fc37cd63f95-kube-api-access-pzc29\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:08.813726 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:08.813574 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6538808b-2145-4538-9620-6fc37cd63f95-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:09.554562 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:09.554537 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-j2zsg" Oct 31 00:07:09.554562 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:09.554555 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-j2zsg" event={"ID":"6538808b-2145-4538-9620-6fc37cd63f95","Type":"ContainerDied","Data":"0b7cf9d60da3342dcfe917a4eed27df517715b86ea079627a2a09d10d9520c27"} Oct 31 00:07:09.555078 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:09.554604 5274 scope.go:122] "RemoveContainer" containerID="5cb5af950ad6ca30fca2549a3924bf01809af888088b8e35a189b1fdf9a4f123" Oct 31 00:07:09.572034 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:09.572008 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-58bbdb5dcb-j2zsg"] Oct 31 00:07:09.577821 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:09.577800 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-58bbdb5dcb-j2zsg"] Oct 31 00:07:11.481072 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:11.481042 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6538808b-2145-4538-9620-6fc37cd63f95" path="/var/lib/kubelet/pods/6538808b-2145-4538-9620-6fc37cd63f95/volumes" Oct 31 00:07:11.791701 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:11.791631 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5c7449969d-875dj"] Oct 31 00:07:11.792002 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:11.791988 5274 cpu_manager.go:414] "containerMap: removing container" podUID="6538808b-2145-4538-9620-6fc37cd63f95" containerName="korrel8r" Oct 31 00:07:11.792064 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:11.792003 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6538808b-2145-4538-9620-6fc37cd63f95" containerName="korrel8r" Oct 31 00:07:11.792064 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:11.792056 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="6538808b-2145-4538-9620-6fc37cd63f95" containerName="korrel8r" Oct 31 00:07:11.798131 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:11.798108 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5c7449969d-875dj" Oct 31 00:07:11.808751 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:11.808725 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:07:11.808751 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:11.808727 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:07:11.808899 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:11.808762 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 31 00:07:11.808899 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:11.808762 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 31 00:07:11.808899 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:11.808846 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-p9wn4\"" Oct 31 00:07:11.813259 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:11.813240 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5c7449969d-875dj"] Oct 31 00:07:11.938663 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:11.938630 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a1cf23cb-bcf6-4896-b37e-86b050ecd148-config\") pod \"korrel8r-5c7449969d-875dj\" (UID: \"a1cf23cb-bcf6-4896-b37e-86b050ecd148\") " pod="redhat-ods-monitoring/korrel8r-5c7449969d-875dj" Oct 31 00:07:11.938802 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:11.938703 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/a1cf23cb-bcf6-4896-b37e-86b050ecd148-tls\") pod \"korrel8r-5c7449969d-875dj\" (UID: \"a1cf23cb-bcf6-4896-b37e-86b050ecd148\") " pod="redhat-ods-monitoring/korrel8r-5c7449969d-875dj" Oct 31 00:07:11.938802 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:11.938762 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dds9m\" (UniqueName: \"kubernetes.io/projected/a1cf23cb-bcf6-4896-b37e-86b050ecd148-kube-api-access-dds9m\") pod \"korrel8r-5c7449969d-875dj\" (UID: \"a1cf23cb-bcf6-4896-b37e-86b050ecd148\") " pod="redhat-ods-monitoring/korrel8r-5c7449969d-875dj" Oct 31 00:07:12.039965 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:12.039914 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a1cf23cb-bcf6-4896-b37e-86b050ecd148-config\") pod \"korrel8r-5c7449969d-875dj\" (UID: \"a1cf23cb-bcf6-4896-b37e-86b050ecd148\") " pod="redhat-ods-monitoring/korrel8r-5c7449969d-875dj" Oct 31 00:07:12.040101 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:12.040010 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/a1cf23cb-bcf6-4896-b37e-86b050ecd148-tls\") pod \"korrel8r-5c7449969d-875dj\" (UID: \"a1cf23cb-bcf6-4896-b37e-86b050ecd148\") " pod="redhat-ods-monitoring/korrel8r-5c7449969d-875dj" Oct 31 00:07:12.040101 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:12.040037 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dds9m\" (UniqueName: \"kubernetes.io/projected/a1cf23cb-bcf6-4896-b37e-86b050ecd148-kube-api-access-dds9m\") pod \"korrel8r-5c7449969d-875dj\" (UID: \"a1cf23cb-bcf6-4896-b37e-86b050ecd148\") " pod="redhat-ods-monitoring/korrel8r-5c7449969d-875dj" Oct 31 00:07:12.040621 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:12.040600 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a1cf23cb-bcf6-4896-b37e-86b050ecd148-config\") pod \"korrel8r-5c7449969d-875dj\" (UID: \"a1cf23cb-bcf6-4896-b37e-86b050ecd148\") " pod="redhat-ods-monitoring/korrel8r-5c7449969d-875dj" Oct 31 00:07:12.042504 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:12.042454 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/a1cf23cb-bcf6-4896-b37e-86b050ecd148-tls\") pod \"korrel8r-5c7449969d-875dj\" (UID: \"a1cf23cb-bcf6-4896-b37e-86b050ecd148\") " pod="redhat-ods-monitoring/korrel8r-5c7449969d-875dj" Oct 31 00:07:12.052461 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:12.052441 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dds9m\" (UniqueName: \"kubernetes.io/projected/a1cf23cb-bcf6-4896-b37e-86b050ecd148-kube-api-access-dds9m\") pod \"korrel8r-5c7449969d-875dj\" (UID: \"a1cf23cb-bcf6-4896-b37e-86b050ecd148\") " pod="redhat-ods-monitoring/korrel8r-5c7449969d-875dj" Oct 31 00:07:12.111473 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:12.111434 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5c7449969d-875dj" Oct 31 00:07:12.400312 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:12.400287 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5c7449969d-875dj"] Oct 31 00:07:12.402630 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:07:12.402603 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1cf23cb_bcf6_4896_b37e_86b050ecd148.slice/crio-ff9423000d5696695e6d19da04c7e5330e011fc8ce52a07b1a8234c7f1b551ab WatchSource:0}: Error finding container ff9423000d5696695e6d19da04c7e5330e011fc8ce52a07b1a8234c7f1b551ab: Status 404 returned error can't find the container with id ff9423000d5696695e6d19da04c7e5330e011fc8ce52a07b1a8234c7f1b551ab Oct 31 00:07:12.578590 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:12.578556 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5c7449969d-875dj" event={"ID":"a1cf23cb-bcf6-4896-b37e-86b050ecd148","Type":"ContainerStarted","Data":"f0f06ff3b5f26f598ee18e2996569be7d4088d6db83a1ddcefc73471c5301202"} Oct 31 00:07:12.578985 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:12.578642 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5c7449969d-875dj" event={"ID":"a1cf23cb-bcf6-4896-b37e-86b050ecd148","Type":"ContainerStarted","Data":"ff9423000d5696695e6d19da04c7e5330e011fc8ce52a07b1a8234c7f1b551ab"} Oct 31 00:07:12.578985 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:12.578762 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5c7449969d-875dj" Oct 31 00:07:12.597677 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:12.597631 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5c7449969d-875dj" podStartSLOduration=1.5976193859999999 podStartE2EDuration="1.597619386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:07:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:07:12.59626284 +0000 UTC m=+469.233081559" watchObservedRunningTime="2026-10-31 00:07:12.597619386 +0000 UTC m=+469.234438102" Oct 31 00:07:20.898200 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:07:20.898156 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:07:23.585495 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:23.585467 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5c7449969d-875dj" Oct 31 00:07:25.202972 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:25.202921 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5c7449969d-875dj"] Oct 31 00:07:25.203351 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:25.203115 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5c7449969d-875dj" podUID="a1cf23cb-bcf6-4896-b37e-86b050ecd148" containerName="korrel8r" containerID="cri-o://f0f06ff3b5f26f598ee18e2996569be7d4088d6db83a1ddcefc73471c5301202" gracePeriod=30 Oct 31 00:07:25.471140 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:25.471066 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5c7449969d-875dj" Oct 31 00:07:25.550400 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:25.550371 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1cf23cb-bcf6-4896-b37e-86b050ecd148-kube-api-access-dds9m\" (UniqueName: \"kubernetes.io/projected/a1cf23cb-bcf6-4896-b37e-86b050ecd148-kube-api-access-dds9m\") pod \"a1cf23cb-bcf6-4896-b37e-86b050ecd148\" (UID: \"a1cf23cb-bcf6-4896-b37e-86b050ecd148\") " Oct 31 00:07:25.550594 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:25.550444 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1cf23cb-bcf6-4896-b37e-86b050ecd148-config\" (UniqueName: \"kubernetes.io/configmap/a1cf23cb-bcf6-4896-b37e-86b050ecd148-config\") pod \"a1cf23cb-bcf6-4896-b37e-86b050ecd148\" (UID: \"a1cf23cb-bcf6-4896-b37e-86b050ecd148\") " Oct 31 00:07:25.550594 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:25.550467 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1cf23cb-bcf6-4896-b37e-86b050ecd148-tls\" (UniqueName: \"kubernetes.io/secret/a1cf23cb-bcf6-4896-b37e-86b050ecd148-tls\") pod \"a1cf23cb-bcf6-4896-b37e-86b050ecd148\" (UID: \"a1cf23cb-bcf6-4896-b37e-86b050ecd148\") " Oct 31 00:07:25.550948 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:25.550919 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1cf23cb-bcf6-4896-b37e-86b050ecd148-config" pod "a1cf23cb-bcf6-4896-b37e-86b050ecd148" (UID: "a1cf23cb-bcf6-4896-b37e-86b050ecd148"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:07:25.552647 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:25.552608 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1cf23cb-bcf6-4896-b37e-86b050ecd148-kube-api-access-dds9m" pod "a1cf23cb-bcf6-4896-b37e-86b050ecd148" (UID: "a1cf23cb-bcf6-4896-b37e-86b050ecd148"). InnerVolumeSpecName "kube-api-access-dds9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:07:25.552764 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:25.552709 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1cf23cb-bcf6-4896-b37e-86b050ecd148-tls" pod "a1cf23cb-bcf6-4896-b37e-86b050ecd148" (UID: "a1cf23cb-bcf6-4896-b37e-86b050ecd148"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:07:25.647133 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:25.647099 5274 generic.go:356] "Generic (PLEG): container finished" podID="a1cf23cb-bcf6-4896-b37e-86b050ecd148" containerID="f0f06ff3b5f26f598ee18e2996569be7d4088d6db83a1ddcefc73471c5301202" exitCode=0 Oct 31 00:07:25.647310 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:25.647155 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5c7449969d-875dj" Oct 31 00:07:25.647310 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:25.647187 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5c7449969d-875dj" event={"ID":"a1cf23cb-bcf6-4896-b37e-86b050ecd148","Type":"ContainerDied","Data":"f0f06ff3b5f26f598ee18e2996569be7d4088d6db83a1ddcefc73471c5301202"} Oct 31 00:07:25.647310 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:25.647257 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5c7449969d-875dj" event={"ID":"a1cf23cb-bcf6-4896-b37e-86b050ecd148","Type":"ContainerDied","Data":"ff9423000d5696695e6d19da04c7e5330e011fc8ce52a07b1a8234c7f1b551ab"} Oct 31 00:07:25.647310 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:25.647279 5274 scope.go:122] "RemoveContainer" containerID="f0f06ff3b5f26f598ee18e2996569be7d4088d6db83a1ddcefc73471c5301202" Oct 31 00:07:25.651252 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:25.651234 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a1cf23cb-bcf6-4896-b37e-86b050ecd148-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:25.651345 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:25.651255 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/a1cf23cb-bcf6-4896-b37e-86b050ecd148-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:25.651345 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:25.651268 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dds9m\" (UniqueName: \"kubernetes.io/projected/a1cf23cb-bcf6-4896-b37e-86b050ecd148-kube-api-access-dds9m\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:25.655909 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:25.655895 5274 scope.go:122] "RemoveContainer" containerID="f0f06ff3b5f26f598ee18e2996569be7d4088d6db83a1ddcefc73471c5301202" Oct 31 00:07:25.656182 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:07:25.656164 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0f06ff3b5f26f598ee18e2996569be7d4088d6db83a1ddcefc73471c5301202\": container with ID starting with f0f06ff3b5f26f598ee18e2996569be7d4088d6db83a1ddcefc73471c5301202 not found: ID does not exist" containerID="f0f06ff3b5f26f598ee18e2996569be7d4088d6db83a1ddcefc73471c5301202" Oct 31 00:07:25.656253 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:25.656187 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0f06ff3b5f26f598ee18e2996569be7d4088d6db83a1ddcefc73471c5301202"} err="failed to get container status \"f0f06ff3b5f26f598ee18e2996569be7d4088d6db83a1ddcefc73471c5301202\": rpc error: code = NotFound desc = could not find container \"f0f06ff3b5f26f598ee18e2996569be7d4088d6db83a1ddcefc73471c5301202\": container with ID starting with f0f06ff3b5f26f598ee18e2996569be7d4088d6db83a1ddcefc73471c5301202 not found: ID does not exist" Oct 31 00:07:25.669792 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:25.669771 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5c7449969d-875dj"] Oct 31 00:07:25.673797 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:25.673778 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5c7449969d-875dj"] Oct 31 00:07:27.481738 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:27.481700 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1cf23cb-bcf6-4896-b37e-86b050ecd148" path="/var/lib/kubelet/pods/a1cf23cb-bcf6-4896-b37e-86b050ecd148/volumes" Oct 31 00:07:33.065402 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.065369 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nbzfw"] Oct 31 00:07:33.065768 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.065713 5274 cpu_manager.go:414] "containerMap: removing container" podUID="a1cf23cb-bcf6-4896-b37e-86b050ecd148" containerName="korrel8r" Oct 31 00:07:33.065768 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.065724 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1cf23cb-bcf6-4896-b37e-86b050ecd148" containerName="korrel8r" Oct 31 00:07:33.065843 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.065804 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1cf23cb-bcf6-4896-b37e-86b050ecd148" containerName="korrel8r" Oct 31 00:07:33.070194 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.070171 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nbzfw" Oct 31 00:07:33.088135 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.088113 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:07:33.088272 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.088207 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 31 00:07:33.088343 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.088279 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-g7v6s\"" Oct 31 00:07:33.088343 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.088320 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:07:33.088457 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.088367 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\"" Oct 31 00:07:33.088457 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.088394 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\"" Oct 31 00:07:33.088457 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.088276 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:07:33.101026 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.101006 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nbzfw"] Oct 31 00:07:33.166519 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.166489 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-gvvck"] Oct 31 00:07:33.170041 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.170020 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gvvck" Oct 31 00:07:33.181347 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.181326 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-gvvck"] Oct 31 00:07:33.187139 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.187120 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 31 00:07:33.188246 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.188216 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-pvfxs\"" Oct 31 00:07:33.188358 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.188266 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 31 00:07:33.216815 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.216795 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec56f3ec-c11e-4358-8707-6723210bbebc-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-nbzfw\" (UID: \"ec56f3ec-c11e-4358-8707-6723210bbebc\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nbzfw" Oct 31 00:07:33.216898 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.216858 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ec56f3ec-c11e-4358-8707-6723210bbebc-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-nbzfw\" (UID: \"ec56f3ec-c11e-4358-8707-6723210bbebc\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nbzfw" Oct 31 00:07:33.216939 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.216921 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vkxq\" (UniqueName: \"kubernetes.io/projected/ec56f3ec-c11e-4358-8707-6723210bbebc-kube-api-access-9vkxq\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-nbzfw\" (UID: \"ec56f3ec-c11e-4358-8707-6723210bbebc\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nbzfw" Oct 31 00:07:33.216988 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.216943 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/ec56f3ec-c11e-4358-8707-6723210bbebc-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-nbzfw\" (UID: \"ec56f3ec-c11e-4358-8707-6723210bbebc\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nbzfw" Oct 31 00:07:33.217098 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.217075 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/ec56f3ec-c11e-4358-8707-6723210bbebc-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-nbzfw\" (UID: \"ec56f3ec-c11e-4358-8707-6723210bbebc\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nbzfw" Oct 31 00:07:33.318211 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.318148 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec56f3ec-c11e-4358-8707-6723210bbebc-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-nbzfw\" (UID: \"ec56f3ec-c11e-4358-8707-6723210bbebc\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nbzfw" Oct 31 00:07:33.318211 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.318188 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjh8f\" (UniqueName: \"kubernetes.io/projected/b041f16e-c90d-4083-b5ab-5a52412e739a-kube-api-access-fjh8f\") pod \"korrel8r-5cbcc89b8f-gvvck\" (UID: \"b041f16e-c90d-4083-b5ab-5a52412e739a\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gvvck" Oct 31 00:07:33.318362 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.318214 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b041f16e-c90d-4083-b5ab-5a52412e739a-tls\") pod \"korrel8r-5cbcc89b8f-gvvck\" (UID: \"b041f16e-c90d-4083-b5ab-5a52412e739a\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gvvck" Oct 31 00:07:33.318362 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.318325 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ec56f3ec-c11e-4358-8707-6723210bbebc-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-nbzfw\" (UID: \"ec56f3ec-c11e-4358-8707-6723210bbebc\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nbzfw" Oct 31 00:07:33.318362 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.318348 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vkxq\" (UniqueName: \"kubernetes.io/projected/ec56f3ec-c11e-4358-8707-6723210bbebc-kube-api-access-9vkxq\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-nbzfw\" (UID: \"ec56f3ec-c11e-4358-8707-6723210bbebc\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nbzfw" Oct 31 00:07:33.318503 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.318365 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/ec56f3ec-c11e-4358-8707-6723210bbebc-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-nbzfw\" (UID: \"ec56f3ec-c11e-4358-8707-6723210bbebc\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nbzfw" Oct 31 00:07:33.318503 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.318399 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/ec56f3ec-c11e-4358-8707-6723210bbebc-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-nbzfw\" (UID: \"ec56f3ec-c11e-4358-8707-6723210bbebc\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nbzfw" Oct 31 00:07:33.318503 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.318435 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b041f16e-c90d-4083-b5ab-5a52412e739a-config\") pod \"korrel8r-5cbcc89b8f-gvvck\" (UID: \"b041f16e-c90d-4083-b5ab-5a52412e739a\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gvvck" Oct 31 00:07:33.319216 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.319190 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/ec56f3ec-c11e-4358-8707-6723210bbebc-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-nbzfw\" (UID: \"ec56f3ec-c11e-4358-8707-6723210bbebc\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nbzfw" Oct 31 00:07:33.321493 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.321474 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec56f3ec-c11e-4358-8707-6723210bbebc-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-nbzfw\" (UID: \"ec56f3ec-c11e-4358-8707-6723210bbebc\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nbzfw" Oct 31 00:07:33.321587 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.321538 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/ec56f3ec-c11e-4358-8707-6723210bbebc-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-nbzfw\" (UID: \"ec56f3ec-c11e-4358-8707-6723210bbebc\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nbzfw" Oct 31 00:07:33.324435 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.324389 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ec56f3ec-c11e-4358-8707-6723210bbebc-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-nbzfw\" (UID: \"ec56f3ec-c11e-4358-8707-6723210bbebc\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nbzfw" Oct 31 00:07:33.336836 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.336811 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vkxq\" (UniqueName: \"kubernetes.io/projected/ec56f3ec-c11e-4358-8707-6723210bbebc-kube-api-access-9vkxq\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-nbzfw\" (UID: \"ec56f3ec-c11e-4358-8707-6723210bbebc\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nbzfw" Oct 31 00:07:33.382550 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.382522 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nbzfw" Oct 31 00:07:33.419436 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.419401 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjh8f\" (UniqueName: \"kubernetes.io/projected/b041f16e-c90d-4083-b5ab-5a52412e739a-kube-api-access-fjh8f\") pod \"korrel8r-5cbcc89b8f-gvvck\" (UID: \"b041f16e-c90d-4083-b5ab-5a52412e739a\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gvvck" Oct 31 00:07:33.419552 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.419444 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b041f16e-c90d-4083-b5ab-5a52412e739a-tls\") pod \"korrel8r-5cbcc89b8f-gvvck\" (UID: \"b041f16e-c90d-4083-b5ab-5a52412e739a\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gvvck" Oct 31 00:07:33.419552 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.419497 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b041f16e-c90d-4083-b5ab-5a52412e739a-config\") pod \"korrel8r-5cbcc89b8f-gvvck\" (UID: \"b041f16e-c90d-4083-b5ab-5a52412e739a\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gvvck" Oct 31 00:07:33.420216 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.420194 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b041f16e-c90d-4083-b5ab-5a52412e739a-config\") pod \"korrel8r-5cbcc89b8f-gvvck\" (UID: \"b041f16e-c90d-4083-b5ab-5a52412e739a\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gvvck" Oct 31 00:07:33.422002 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.421915 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b041f16e-c90d-4083-b5ab-5a52412e739a-tls\") pod \"korrel8r-5cbcc89b8f-gvvck\" (UID: \"b041f16e-c90d-4083-b5ab-5a52412e739a\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gvvck" Oct 31 00:07:33.432869 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.432841 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjh8f\" (UniqueName: \"kubernetes.io/projected/b041f16e-c90d-4083-b5ab-5a52412e739a-kube-api-access-fjh8f\") pod \"korrel8r-5cbcc89b8f-gvvck\" (UID: \"b041f16e-c90d-4083-b5ab-5a52412e739a\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gvvck" Oct 31 00:07:33.483625 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.483595 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gvvck" Oct 31 00:07:33.678486 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.678452 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nbzfw"] Oct 31 00:07:33.680945 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:07:33.680750 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec56f3ec_c11e_4358_8707_6723210bbebc.slice/crio-a6cf6e946b07fcbfc1f830722cc6bf588ed02d1784fe0f36952aacaa78ee26cc WatchSource:0}: Error finding container a6cf6e946b07fcbfc1f830722cc6bf588ed02d1784fe0f36952aacaa78ee26cc: Status 404 returned error can't find the container with id a6cf6e946b07fcbfc1f830722cc6bf588ed02d1784fe0f36952aacaa78ee26cc Oct 31 00:07:33.690686 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.690657 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nbzfw" event={"ID":"ec56f3ec-c11e-4358-8707-6723210bbebc","Type":"ContainerStarted","Data":"a6cf6e946b07fcbfc1f830722cc6bf588ed02d1784fe0f36952aacaa78ee26cc"} Oct 31 00:07:33.762802 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:33.762775 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-gvvck"] Oct 31 00:07:33.764660 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:07:33.764630 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb041f16e_c90d_4083_b5ab_5a52412e739a.slice/crio-de42ec479c00f7c9344cf229979d1f8fda4bf20c093d281fef63dd8a2cb0556b WatchSource:0}: Error finding container de42ec479c00f7c9344cf229979d1f8fda4bf20c093d281fef63dd8a2cb0556b: Status 404 returned error can't find the container with id de42ec479c00f7c9344cf229979d1f8fda4bf20c093d281fef63dd8a2cb0556b Oct 31 00:07:34.699361 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:34.699324 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gvvck" event={"ID":"b041f16e-c90d-4083-b5ab-5a52412e739a","Type":"ContainerStarted","Data":"0894ddc44b09a8d4c280c05e90203133aac8f3b7a355ab48f34684cfa5c3e041"} Oct 31 00:07:34.699804 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:34.699393 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gvvck" event={"ID":"b041f16e-c90d-4083-b5ab-5a52412e739a","Type":"ContainerStarted","Data":"de42ec479c00f7c9344cf229979d1f8fda4bf20c093d281fef63dd8a2cb0556b"} Oct 31 00:07:34.699804 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:34.699447 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gvvck" Oct 31 00:07:34.721390 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:34.721327 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gvvck" podStartSLOduration=1.721313444 podStartE2EDuration="1.721313444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:07:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:07:34.720179111 +0000 UTC m=+491.356997827" watchObservedRunningTime="2026-10-31 00:07:34.721313444 +0000 UTC m=+491.358132160" Oct 31 00:07:37.717000 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:37.716945 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nbzfw" event={"ID":"ec56f3ec-c11e-4358-8707-6723210bbebc","Type":"ContainerStarted","Data":"448e85686ef9d9ac38feb4d6310dc88bd700706e235512a056967dadc4bb6473"} Oct 31 00:07:37.717464 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:37.717146 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nbzfw" Oct 31 00:07:37.739826 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:37.739783 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nbzfw" podStartSLOduration=1.7923731 podStartE2EDuration="4.739770634s" totalImagesPullingTime="2.947397534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:07:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:07:37.737530464 +0000 UTC m=+494.374349184" watchObservedRunningTime="2026-10-31 00:07:37.739770634 +0000 UTC m=+494.376589350" Oct 31 00:07:40.475208 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:40.475172 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:07:40.485841 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:40.485815 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:07:40.497789 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:40.497768 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 31 00:07:40.500304 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:40.500279 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:07:40.501103 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:40.500565 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 31 00:07:40.501103 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:40.500588 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:07:40.501292 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:40.500622 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 31 00:07:40.501292 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:40.500720 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-8tt9h\"" Oct 31 00:07:40.581647 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:40.581617 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/a99ee414-acb3-4091-9e6d-825a4c76a2e4-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"a99ee414-acb3-4091-9e6d-825a4c76a2e4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:07:40.581774 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:40.581664 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a99ee414-acb3-4091-9e6d-825a4c76a2e4-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"a99ee414-acb3-4091-9e6d-825a4c76a2e4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:07:40.581774 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:40.581725 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a99ee414-acb3-4091-9e6d-825a4c76a2e4-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"a99ee414-acb3-4091-9e6d-825a4c76a2e4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:07:40.581849 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:40.581820 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a99ee414-acb3-4091-9e6d-825a4c76a2e4-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"a99ee414-acb3-4091-9e6d-825a4c76a2e4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:07:40.581883 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:40.581861 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a99ee414-acb3-4091-9e6d-825a4c76a2e4-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"a99ee414-acb3-4091-9e6d-825a4c76a2e4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:07:40.581916 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:40.581882 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfc5z\" (UniqueName: \"kubernetes.io/projected/a99ee414-acb3-4091-9e6d-825a4c76a2e4-kube-api-access-mfc5z\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"a99ee414-acb3-4091-9e6d-825a4c76a2e4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:07:40.581984 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:40.581932 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a99ee414-acb3-4091-9e6d-825a4c76a2e4-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"a99ee414-acb3-4091-9e6d-825a4c76a2e4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:07:40.683027 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:40.682992 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfc5z\" (UniqueName: \"kubernetes.io/projected/a99ee414-acb3-4091-9e6d-825a4c76a2e4-kube-api-access-mfc5z\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"a99ee414-acb3-4091-9e6d-825a4c76a2e4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:07:40.683192 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:40.683044 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a99ee414-acb3-4091-9e6d-825a4c76a2e4-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"a99ee414-acb3-4091-9e6d-825a4c76a2e4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:07:40.683192 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:40.683072 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/a99ee414-acb3-4091-9e6d-825a4c76a2e4-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"a99ee414-acb3-4091-9e6d-825a4c76a2e4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:07:40.683192 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:40.683128 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a99ee414-acb3-4091-9e6d-825a4c76a2e4-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"a99ee414-acb3-4091-9e6d-825a4c76a2e4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:07:40.683192 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:40.683173 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a99ee414-acb3-4091-9e6d-825a4c76a2e4-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"a99ee414-acb3-4091-9e6d-825a4c76a2e4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:07:40.683407 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:40.683217 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a99ee414-acb3-4091-9e6d-825a4c76a2e4-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"a99ee414-acb3-4091-9e6d-825a4c76a2e4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:07:40.683407 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:40.683266 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a99ee414-acb3-4091-9e6d-825a4c76a2e4-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"a99ee414-acb3-4091-9e6d-825a4c76a2e4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:07:40.683606 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:40.683576 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/a99ee414-acb3-4091-9e6d-825a4c76a2e4-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"a99ee414-acb3-4091-9e6d-825a4c76a2e4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:07:40.685426 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:40.685393 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a99ee414-acb3-4091-9e6d-825a4c76a2e4-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"a99ee414-acb3-4091-9e6d-825a4c76a2e4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:07:40.685602 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:40.685582 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a99ee414-acb3-4091-9e6d-825a4c76a2e4-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"a99ee414-acb3-4091-9e6d-825a4c76a2e4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:07:40.685738 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:40.685718 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a99ee414-acb3-4091-9e6d-825a4c76a2e4-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"a99ee414-acb3-4091-9e6d-825a4c76a2e4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:07:40.685868 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:40.685853 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a99ee414-acb3-4091-9e6d-825a4c76a2e4-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"a99ee414-acb3-4091-9e6d-825a4c76a2e4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:07:40.686095 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:40.686075 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a99ee414-acb3-4091-9e6d-825a4c76a2e4-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"a99ee414-acb3-4091-9e6d-825a4c76a2e4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:07:40.695515 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:40.695495 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfc5z\" (UniqueName: \"kubernetes.io/projected/a99ee414-acb3-4091-9e6d-825a4c76a2e4-kube-api-access-mfc5z\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"a99ee414-acb3-4091-9e6d-825a4c76a2e4\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:07:40.812336 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:40.812271 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:07:41.091757 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:41.091730 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:07:41.093300 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:07:41.093272 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda99ee414_acb3_4091_9e6d_825a4c76a2e4.slice/crio-17d8754a73bd98a726da76d0bfa2adac4658aeda608d38503bcebb0acb537229 WatchSource:0}: Error finding container 17d8754a73bd98a726da76d0bfa2adac4658aeda608d38503bcebb0acb537229: Status 404 returned error can't find the container with id 17d8754a73bd98a726da76d0bfa2adac4658aeda608d38503bcebb0acb537229 Oct 31 00:07:41.738297 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:41.738263 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"a99ee414-acb3-4091-9e6d-825a4c76a2e4","Type":"ContainerStarted","Data":"17d8754a73bd98a726da76d0bfa2adac4658aeda608d38503bcebb0acb537229"} Oct 31 00:07:43.515695 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.515664 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:07:43.519577 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.519561 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:43.525600 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.525573 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-qtj4l\"" Oct 31 00:07:43.525744 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.525633 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:07:43.525744 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.525638 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 31 00:07:43.525744 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.525672 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:07:43.526116 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.525916 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 31 00:07:43.526168 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.526115 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 31 00:07:43.527018 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.526989 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 31 00:07:43.527018 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.527005 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 31 00:07:43.527165 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.527103 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 31 00:07:43.537315 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.537291 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:07:43.610153 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.610070 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9920297-9a26-49bd-bb8f-1900f94074e8-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:43.610153 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.610115 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9920297-9a26-49bd-bb8f-1900f94074e8-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:43.610342 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.610186 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f9920297-9a26-49bd-bb8f-1900f94074e8-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:43.610342 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.610213 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f9920297-9a26-49bd-bb8f-1900f94074e8-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:43.610342 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.610285 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:43.610342 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.610309 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:43.610472 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.610340 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:43.610472 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.610374 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:43.610472 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.610435 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f9920297-9a26-49bd-bb8f-1900f94074e8-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:43.610472 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.610462 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8vxf\" (UniqueName: \"kubernetes.io/projected/f9920297-9a26-49bd-bb8f-1900f94074e8-kube-api-access-j8vxf\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:43.610640 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.610487 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:43.610640 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.610549 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:43.710965 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.710919 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:43.711158 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.711009 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9920297-9a26-49bd-bb8f-1900f94074e8-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:43.711158 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.711062 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9920297-9a26-49bd-bb8f-1900f94074e8-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:43.711158 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.711109 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f9920297-9a26-49bd-bb8f-1900f94074e8-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:43.711158 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.711140 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f9920297-9a26-49bd-bb8f-1900f94074e8-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:43.711374 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.711181 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:43.711374 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.711199 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:43.711374 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.711233 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:43.711374 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.711249 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:43.711374 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.711267 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f9920297-9a26-49bd-bb8f-1900f94074e8-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:43.711374 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.711292 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8vxf\" (UniqueName: \"kubernetes.io/projected/f9920297-9a26-49bd-bb8f-1900f94074e8-kube-api-access-j8vxf\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:43.711374 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.711321 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:43.712371 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.712301 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f9920297-9a26-49bd-bb8f-1900f94074e8-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:43.712640 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.712609 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f9920297-9a26-49bd-bb8f-1900f94074e8-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:43.712735 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.712673 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f9920297-9a26-49bd-bb8f-1900f94074e8-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:43.714244 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.714209 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9920297-9a26-49bd-bb8f-1900f94074e8-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:43.714393 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.714369 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:43.714924 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.714885 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:43.715060 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.715016 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:43.715331 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.715306 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:43.715751 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.715725 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:43.716699 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.716675 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:43.716798 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.716736 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9920297-9a26-49bd-bb8f-1900f94074e8-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:43.724715 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.724695 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nbzfw" Oct 31 00:07:43.724816 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.724734 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8vxf\" (UniqueName: \"kubernetes.io/projected/f9920297-9a26-49bd-bb8f-1900f94074e8-kube-api-access-j8vxf\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:43.751464 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:43.751427 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"a99ee414-acb3-4091-9e6d-825a4c76a2e4","Type":"ContainerStarted","Data":"1c68e1e44c8cc260672f41b0619ceecb24035b6ccec494b8a4c3d8dc21b22df6"} Oct 31 00:07:45.707726 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:45.707698 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gvvck" Oct 31 00:07:45.931712 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:45.931678 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:45.938724 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:45.938693 5274 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:46.032184 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:46.032108 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:46.032307 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:46.032242 5274 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:46.036173 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:46.036155 5274 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") DevicePath \"csi-51e8027f163b5f98040faf3f8c30848dd44a7a90ca2bd0af2446ea6398ed018d\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:46.201643 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:46.201618 5274 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/53bcf0e8434a816dcc013cea20657add42b4da36ecdc0fff46af7413bc0037a5/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:46.215020 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:46.214998 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:46.239261 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:46.239238 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:07:46.529585 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:46.529529 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:07:46.531303 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:07:46.531272 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9920297_9a26_49bd_bb8f_1900f94074e8.slice/crio-06e12376658b8bb616064eadcc213487cf98d02a60552dc05d45d5a8596482c9 WatchSource:0}: Error finding container 06e12376658b8bb616064eadcc213487cf98d02a60552dc05d45d5a8596482c9: Status 404 returned error can't find the container with id 06e12376658b8bb616064eadcc213487cf98d02a60552dc05d45d5a8596482c9 Oct 31 00:07:46.766641 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:46.766612 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"f9920297-9a26-49bd-bb8f-1900f94074e8","Type":"ContainerStarted","Data":"06e12376658b8bb616064eadcc213487cf98d02a60552dc05d45d5a8596482c9"} Oct 31 00:07:47.772999 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:47.772968 5274 generic.go:356] "Generic (PLEG): container finished" podID="a99ee414-acb3-4091-9e6d-825a4c76a2e4" containerID="1c68e1e44c8cc260672f41b0619ceecb24035b6ccec494b8a4c3d8dc21b22df6" exitCode=0 Oct 31 00:07:47.773423 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:47.773026 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"a99ee414-acb3-4091-9e6d-825a4c76a2e4","Type":"ContainerDied","Data":"1c68e1e44c8cc260672f41b0619ceecb24035b6ccec494b8a4c3d8dc21b22df6"} Oct 31 00:07:47.774670 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:47.774647 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"f9920297-9a26-49bd-bb8f-1900f94074e8","Type":"ContainerStarted","Data":"2fb428f880e65496e654d888ceea1168a1f6de2d4b18986ba8600ac51ce8b44e"} Oct 31 00:07:50.792628 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:50.792595 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"a99ee414-acb3-4091-9e6d-825a4c76a2e4","Type":"ContainerStarted","Data":"e312b7a1fabcb037d9ca35380ecec359f22af00646bbbd13ac0eb5cdef666bff"} Oct 31 00:07:50.793004 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:50.792663 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"a99ee414-acb3-4091-9e6d-825a4c76a2e4","Type":"ContainerStarted","Data":"4a788c4f910fa7e357a3e1c5e36e87cb0b34c8cbd606f2a1a16dace07620ddce"} Oct 31 00:07:50.793004 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:50.792862 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:07:50.794314 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:50.794291 5274 generic.go:356] "Generic (PLEG): container finished" podID="f9920297-9a26-49bd-bb8f-1900f94074e8" containerID="2fb428f880e65496e654d888ceea1168a1f6de2d4b18986ba8600ac51ce8b44e" exitCode=0 Oct 31 00:07:50.794406 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:50.794334 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"f9920297-9a26-49bd-bb8f-1900f94074e8","Type":"ContainerDied","Data":"2fb428f880e65496e654d888ceea1168a1f6de2d4b18986ba8600ac51ce8b44e"} Oct 31 00:07:50.826904 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:50.826864 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=2.696872703 podStartE2EDuration="10.826851995s" totalImagesPullingTime="4.444081869s" totalInitContainerRuntime="3.685897423s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:07:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:07:50.825235412 +0000 UTC m=+507.462054128" watchObservedRunningTime="2026-10-31 00:07:50.826851995 +0000 UTC m=+507.463670733" Oct 31 00:07:50.898795 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:07:50.898769 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:07:55.822781 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:55.822701 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"f9920297-9a26-49bd-bb8f-1900f94074e8","Type":"ContainerStarted","Data":"9d38a5b791efb7a0922794fdac8bbde4d4385077a526b14f021892fc87d634a6"} Oct 31 00:07:56.801285 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:56.801252 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:07:56.830021 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:56.829935 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"f9920297-9a26-49bd-bb8f-1900f94074e8","Type":"ContainerStarted","Data":"5e0c57752d1d7afa29ec624f12c2f45abce2e376eccb9f4976d2c62a879f5519"} Oct 31 00:07:59.849305 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:59.849274 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"f9920297-9a26-49bd-bb8f-1900f94074e8","Type":"ContainerStarted","Data":"b063bf2f1aa10d5003a6652b9fc76df68372541201b8a6e2d0252fa2bed46767"} Oct 31 00:07:59.887148 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:07:59.887080 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=8.740241982 podStartE2EDuration="19.887062075s" totalImagesPullingTime="7.959517866s" totalInitContainerRuntime="3.187302227s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:07:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:07:59.884590244 +0000 UTC m=+516.521408961" watchObservedRunningTime="2026-10-31 00:07:59.887062075 +0000 UTC m=+516.523880792" Oct 31 00:08:01.240269 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:01.240232 5274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:08:01.240684 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:01.240325 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:08:01.245587 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:01.245565 5274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:08:01.864240 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:01.864216 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:08:07.147025 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.146992 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nbzfw"] Oct 31 00:08:07.147414 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.147213 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nbzfw" podUID="ec56f3ec-c11e-4358-8707-6723210bbebc" containerName="kube-rbac-proxy" containerID="cri-o://448e85686ef9d9ac38feb4d6310dc88bd700706e235512a056967dadc4bb6473" gracePeriod=30 Oct 31 00:08:07.162776 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.162750 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-gvvck"] Oct 31 00:08:07.162934 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.162916 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gvvck" podUID="b041f16e-c90d-4083-b5ab-5a52412e739a" containerName="korrel8r" containerID="cri-o://0894ddc44b09a8d4c280c05e90203133aac8f3b7a355ab48f34684cfa5c3e041" gracePeriod=30 Oct 31 00:08:07.399946 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.399874 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:08:07.400273 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.400232 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="f9920297-9a26-49bd-bb8f-1900f94074e8" containerName="prometheus" containerID="cri-o://9d38a5b791efb7a0922794fdac8bbde4d4385077a526b14f021892fc87d634a6" gracePeriod=600 Oct 31 00:08:07.400799 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.400628 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="f9920297-9a26-49bd-bb8f-1900f94074e8" containerName="thanos-sidecar" containerID="cri-o://b063bf2f1aa10d5003a6652b9fc76df68372541201b8a6e2d0252fa2bed46767" gracePeriod=600 Oct 31 00:08:07.400799 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.400748 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="f9920297-9a26-49bd-bb8f-1900f94074e8" containerName="config-reloader" containerID="cri-o://5e0c57752d1d7afa29ec624f12c2f45abce2e376eccb9f4976d2c62a879f5519" gracePeriod=600 Oct 31 00:08:07.413993 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.413911 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:08:07.414564 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.414319 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="a99ee414-acb3-4091-9e6d-825a4c76a2e4" containerName="alertmanager" containerID="cri-o://4a788c4f910fa7e357a3e1c5e36e87cb0b34c8cbd606f2a1a16dace07620ddce" gracePeriod=120 Oct 31 00:08:07.414564 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.414488 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="a99ee414-acb3-4091-9e6d-825a4c76a2e4" containerName="config-reloader" containerID="cri-o://e312b7a1fabcb037d9ca35380ecec359f22af00646bbbd13ac0eb5cdef666bff" gracePeriod=120 Oct 31 00:08:07.565841 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.565814 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gvvck" Oct 31 00:08:07.571578 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.571555 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nbzfw" Oct 31 00:08:07.710092 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.710062 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec56f3ec-c11e-4358-8707-6723210bbebc-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/ec56f3ec-c11e-4358-8707-6723210bbebc-prometheus-client-cert\") pod \"ec56f3ec-c11e-4358-8707-6723210bbebc\" (UID: \"ec56f3ec-c11e-4358-8707-6723210bbebc\") " Oct 31 00:08:07.710268 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.710107 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b041f16e-c90d-4083-b5ab-5a52412e739a-kube-api-access-fjh8f\" (UniqueName: \"kubernetes.io/projected/b041f16e-c90d-4083-b5ab-5a52412e739a-kube-api-access-fjh8f\") pod \"b041f16e-c90d-4083-b5ab-5a52412e739a\" (UID: \"b041f16e-c90d-4083-b5ab-5a52412e739a\") " Oct 31 00:08:07.710268 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.710149 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec56f3ec-c11e-4358-8707-6723210bbebc-tls\" (UniqueName: \"kubernetes.io/secret/ec56f3ec-c11e-4358-8707-6723210bbebc-tls\") pod \"ec56f3ec-c11e-4358-8707-6723210bbebc\" (UID: \"ec56f3ec-c11e-4358-8707-6723210bbebc\") " Oct 31 00:08:07.710268 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.710181 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec56f3ec-c11e-4358-8707-6723210bbebc-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec56f3ec-c11e-4358-8707-6723210bbebc-kube-rbac-proxy-config\") pod \"ec56f3ec-c11e-4358-8707-6723210bbebc\" (UID: \"ec56f3ec-c11e-4358-8707-6723210bbebc\") " Oct 31 00:08:07.710268 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.710198 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b041f16e-c90d-4083-b5ab-5a52412e739a-config\" (UniqueName: \"kubernetes.io/configmap/b041f16e-c90d-4083-b5ab-5a52412e739a-config\") pod \"b041f16e-c90d-4083-b5ab-5a52412e739a\" (UID: \"b041f16e-c90d-4083-b5ab-5a52412e739a\") " Oct 31 00:08:07.710546 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.710518 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec56f3ec-c11e-4358-8707-6723210bbebc-kube-api-access-9vkxq\" (UniqueName: \"kubernetes.io/projected/ec56f3ec-c11e-4358-8707-6723210bbebc-kube-api-access-9vkxq\") pod \"ec56f3ec-c11e-4358-8707-6723210bbebc\" (UID: \"ec56f3ec-c11e-4358-8707-6723210bbebc\") " Oct 31 00:08:07.710625 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.710588 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b041f16e-c90d-4083-b5ab-5a52412e739a-tls\" (UniqueName: \"kubernetes.io/secret/b041f16e-c90d-4083-b5ab-5a52412e739a-tls\") pod \"b041f16e-c90d-4083-b5ab-5a52412e739a\" (UID: \"b041f16e-c90d-4083-b5ab-5a52412e739a\") " Oct 31 00:08:07.710625 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.710610 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b041f16e-c90d-4083-b5ab-5a52412e739a-config" pod "b041f16e-c90d-4083-b5ab-5a52412e739a" (UID: "b041f16e-c90d-4083-b5ab-5a52412e739a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:07.710746 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.710727 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec56f3ec-c11e-4358-8707-6723210bbebc-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/ec56f3ec-c11e-4358-8707-6723210bbebc-prometheus-ca\") pod \"ec56f3ec-c11e-4358-8707-6723210bbebc\" (UID: \"ec56f3ec-c11e-4358-8707-6723210bbebc\") " Oct 31 00:08:07.711273 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.711110 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b041f16e-c90d-4083-b5ab-5a52412e739a-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:07.711444 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.711292 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec56f3ec-c11e-4358-8707-6723210bbebc-prometheus-ca" pod "ec56f3ec-c11e-4358-8707-6723210bbebc" (UID: "ec56f3ec-c11e-4358-8707-6723210bbebc"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:07.712732 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.712690 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec56f3ec-c11e-4358-8707-6723210bbebc-tls" pod "ec56f3ec-c11e-4358-8707-6723210bbebc" (UID: "ec56f3ec-c11e-4358-8707-6723210bbebc"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:07.713364 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.713336 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b041f16e-c90d-4083-b5ab-5a52412e739a-kube-api-access-fjh8f" pod "b041f16e-c90d-4083-b5ab-5a52412e739a" (UID: "b041f16e-c90d-4083-b5ab-5a52412e739a"). InnerVolumeSpecName "kube-api-access-fjh8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:07.713364 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.713356 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b041f16e-c90d-4083-b5ab-5a52412e739a-tls" pod "b041f16e-c90d-4083-b5ab-5a52412e739a" (UID: "b041f16e-c90d-4083-b5ab-5a52412e739a"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:07.713520 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.713371 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec56f3ec-c11e-4358-8707-6723210bbebc-kube-api-access-9vkxq" pod "ec56f3ec-c11e-4358-8707-6723210bbebc" (UID: "ec56f3ec-c11e-4358-8707-6723210bbebc"). InnerVolumeSpecName "kube-api-access-9vkxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:07.713769 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.713752 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec56f3ec-c11e-4358-8707-6723210bbebc-kube-rbac-proxy-config" pod "ec56f3ec-c11e-4358-8707-6723210bbebc" (UID: "ec56f3ec-c11e-4358-8707-6723210bbebc"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:07.714578 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.714551 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec56f3ec-c11e-4358-8707-6723210bbebc-prometheus-client-cert" pod "ec56f3ec-c11e-4358-8707-6723210bbebc" (UID: "ec56f3ec-c11e-4358-8707-6723210bbebc"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:07.727397 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.727377 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:08:07.812427 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.812385 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/ec56f3ec-c11e-4358-8707-6723210bbebc-prometheus-ca\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:07.812427 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.812425 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/ec56f3ec-c11e-4358-8707-6723210bbebc-prometheus-client-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:07.812657 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.812446 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjh8f\" (UniqueName: \"kubernetes.io/projected/b041f16e-c90d-4083-b5ab-5a52412e739a-kube-api-access-fjh8f\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:07.812657 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.812462 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ec56f3ec-c11e-4358-8707-6723210bbebc-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:07.812657 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.812476 5274 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec56f3ec-c11e-4358-8707-6723210bbebc-kube-rbac-proxy-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:07.812657 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.812504 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vkxq\" (UniqueName: \"kubernetes.io/projected/ec56f3ec-c11e-4358-8707-6723210bbebc-kube-api-access-9vkxq\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:07.812657 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.812519 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b041f16e-c90d-4083-b5ab-5a52412e739a-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:07.890310 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.890272 5274 generic.go:356] "Generic (PLEG): container finished" podID="b041f16e-c90d-4083-b5ab-5a52412e739a" containerID="0894ddc44b09a8d4c280c05e90203133aac8f3b7a355ab48f34684cfa5c3e041" exitCode=0 Oct 31 00:08:07.890310 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.890319 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gvvck" Oct 31 00:08:07.890551 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.890353 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gvvck" event={"ID":"b041f16e-c90d-4083-b5ab-5a52412e739a","Type":"ContainerDied","Data":"0894ddc44b09a8d4c280c05e90203133aac8f3b7a355ab48f34684cfa5c3e041"} Oct 31 00:08:07.890551 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.890405 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gvvck" event={"ID":"b041f16e-c90d-4083-b5ab-5a52412e739a","Type":"ContainerDied","Data":"de42ec479c00f7c9344cf229979d1f8fda4bf20c093d281fef63dd8a2cb0556b"} Oct 31 00:08:07.890551 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.890432 5274 scope.go:122] "RemoveContainer" containerID="0894ddc44b09a8d4c280c05e90203133aac8f3b7a355ab48f34684cfa5c3e041" Oct 31 00:08:07.893391 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.893370 5274 generic.go:356] "Generic (PLEG): container finished" podID="a99ee414-acb3-4091-9e6d-825a4c76a2e4" containerID="e312b7a1fabcb037d9ca35380ecec359f22af00646bbbd13ac0eb5cdef666bff" exitCode=0 Oct 31 00:08:07.893515 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.893441 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"a99ee414-acb3-4091-9e6d-825a4c76a2e4","Type":"ContainerDied","Data":"e312b7a1fabcb037d9ca35380ecec359f22af00646bbbd13ac0eb5cdef666bff"} Oct 31 00:08:07.895201 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.895034 5274 generic.go:356] "Generic (PLEG): container finished" podID="ec56f3ec-c11e-4358-8707-6723210bbebc" containerID="448e85686ef9d9ac38feb4d6310dc88bd700706e235512a056967dadc4bb6473" exitCode=0 Oct 31 00:08:07.895201 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.895109 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nbzfw" Oct 31 00:08:07.895201 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.895123 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nbzfw" event={"ID":"ec56f3ec-c11e-4358-8707-6723210bbebc","Type":"ContainerDied","Data":"448e85686ef9d9ac38feb4d6310dc88bd700706e235512a056967dadc4bb6473"} Oct 31 00:08:07.895201 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.895173 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nbzfw" event={"ID":"ec56f3ec-c11e-4358-8707-6723210bbebc","Type":"ContainerDied","Data":"a6cf6e946b07fcbfc1f830722cc6bf588ed02d1784fe0f36952aacaa78ee26cc"} Oct 31 00:08:07.901871 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.901852 5274 scope.go:122] "RemoveContainer" containerID="0894ddc44b09a8d4c280c05e90203133aac8f3b7a355ab48f34684cfa5c3e041" Oct 31 00:08:07.902073 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.902050 5274 generic.go:356] "Generic (PLEG): container finished" podID="f9920297-9a26-49bd-bb8f-1900f94074e8" containerID="b063bf2f1aa10d5003a6652b9fc76df68372541201b8a6e2d0252fa2bed46767" exitCode=0 Oct 31 00:08:07.902073 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.902069 5274 generic.go:356] "Generic (PLEG): container finished" podID="f9920297-9a26-49bd-bb8f-1900f94074e8" containerID="5e0c57752d1d7afa29ec624f12c2f45abce2e376eccb9f4976d2c62a879f5519" exitCode=0 Oct 31 00:08:07.902218 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.902078 5274 generic.go:356] "Generic (PLEG): container finished" podID="f9920297-9a26-49bd-bb8f-1900f94074e8" containerID="9d38a5b791efb7a0922794fdac8bbde4d4385077a526b14f021892fc87d634a6" exitCode=0 Oct 31 00:08:07.902218 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.902128 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:08:07.902284 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.902133 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"f9920297-9a26-49bd-bb8f-1900f94074e8","Type":"ContainerDied","Data":"b063bf2f1aa10d5003a6652b9fc76df68372541201b8a6e2d0252fa2bed46767"} Oct 31 00:08:07.902284 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.902267 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"f9920297-9a26-49bd-bb8f-1900f94074e8","Type":"ContainerDied","Data":"5e0c57752d1d7afa29ec624f12c2f45abce2e376eccb9f4976d2c62a879f5519"} Oct 31 00:08:07.902342 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.902295 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"f9920297-9a26-49bd-bb8f-1900f94074e8","Type":"ContainerDied","Data":"9d38a5b791efb7a0922794fdac8bbde4d4385077a526b14f021892fc87d634a6"} Oct 31 00:08:07.902342 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.902317 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"f9920297-9a26-49bd-bb8f-1900f94074e8","Type":"ContainerDied","Data":"06e12376658b8bb616064eadcc213487cf98d02a60552dc05d45d5a8596482c9"} Oct 31 00:08:07.902829 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:08:07.902812 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0894ddc44b09a8d4c280c05e90203133aac8f3b7a355ab48f34684cfa5c3e041\": container with ID starting with 0894ddc44b09a8d4c280c05e90203133aac8f3b7a355ab48f34684cfa5c3e041 not found: ID does not exist" containerID="0894ddc44b09a8d4c280c05e90203133aac8f3b7a355ab48f34684cfa5c3e041" Oct 31 00:08:07.902888 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.902837 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0894ddc44b09a8d4c280c05e90203133aac8f3b7a355ab48f34684cfa5c3e041"} err="failed to get container status \"0894ddc44b09a8d4c280c05e90203133aac8f3b7a355ab48f34684cfa5c3e041\": rpc error: code = NotFound desc = could not find container \"0894ddc44b09a8d4c280c05e90203133aac8f3b7a355ab48f34684cfa5c3e041\": container with ID starting with 0894ddc44b09a8d4c280c05e90203133aac8f3b7a355ab48f34684cfa5c3e041 not found: ID does not exist" Oct 31 00:08:07.902888 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.902852 5274 scope.go:122] "RemoveContainer" containerID="448e85686ef9d9ac38feb4d6310dc88bd700706e235512a056967dadc4bb6473" Oct 31 00:08:07.913785 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.913730 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9920297-9a26-49bd-bb8f-1900f94074e8-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f9920297-9a26-49bd-bb8f-1900f94074e8-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"f9920297-9a26-49bd-bb8f-1900f94074e8\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " Oct 31 00:08:07.913785 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.913777 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9920297-9a26-49bd-bb8f-1900f94074e8-tls-assets\" (UniqueName: \"kubernetes.io/projected/f9920297-9a26-49bd-bb8f-1900f94074e8-tls-assets\") pod \"f9920297-9a26-49bd-bb8f-1900f94074e8\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " Oct 31 00:08:07.913934 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.913806 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-config\" (UniqueName: \"kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-config\") pod \"f9920297-9a26-49bd-bb8f-1900f94074e8\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " Oct 31 00:08:07.913934 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.913841 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"f9920297-9a26-49bd-bb8f-1900f94074e8\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " Oct 31 00:08:07.913934 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.913879 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-thanos-prometheus-http-client-file\") pod \"f9920297-9a26-49bd-bb8f-1900f94074e8\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " Oct 31 00:08:07.914141 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.914121 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-secret-prometheus-web-tls-ca\") pod \"f9920297-9a26-49bd-bb8f-1900f94074e8\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " Oct 31 00:08:07.914141 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.914122 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9920297-9a26-49bd-bb8f-1900f94074e8-prometheus-data-science-monitoringstack-rulefiles-2" pod "f9920297-9a26-49bd-bb8f-1900f94074e8" (UID: "f9920297-9a26-49bd-bb8f-1900f94074e8"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:07.914254 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.914160 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9920297-9a26-49bd-bb8f-1900f94074e8-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f9920297-9a26-49bd-bb8f-1900f94074e8-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"f9920297-9a26-49bd-bb8f-1900f94074e8\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " Oct 31 00:08:07.915036 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.914292 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9920297-9a26-49bd-bb8f-1900f94074e8-config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9920297-9a26-49bd-bb8f-1900f94074e8-config-out\") pod \"f9920297-9a26-49bd-bb8f-1900f94074e8\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " Oct 31 00:08:07.915036 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.914353 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9920297-9a26-49bd-bb8f-1900f94074e8-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f9920297-9a26-49bd-bb8f-1900f94074e8-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"f9920297-9a26-49bd-bb8f-1900f94074e8\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " Oct 31 00:08:07.915036 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.914410 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"f9920297-9a26-49bd-bb8f-1900f94074e8\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " Oct 31 00:08:07.915036 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.914449 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-web-config\" (UniqueName: \"kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-web-config\") pod \"f9920297-9a26-49bd-bb8f-1900f94074e8\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " Oct 31 00:08:07.915036 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.914547 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"f9920297-9a26-49bd-bb8f-1900f94074e8\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " Oct 31 00:08:07.915036 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.914584 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9920297-9a26-49bd-bb8f-1900f94074e8-kube-api-access-j8vxf\" (UniqueName: \"kubernetes.io/projected/f9920297-9a26-49bd-bb8f-1900f94074e8-kube-api-access-j8vxf\") pod \"f9920297-9a26-49bd-bb8f-1900f94074e8\" (UID: \"f9920297-9a26-49bd-bb8f-1900f94074e8\") " Oct 31 00:08:07.915036 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.914727 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9920297-9a26-49bd-bb8f-1900f94074e8-prometheus-data-science-monitoringstack-rulefiles-0" pod "f9920297-9a26-49bd-bb8f-1900f94074e8" (UID: "f9920297-9a26-49bd-bb8f-1900f94074e8"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:07.915036 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.914891 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f9920297-9a26-49bd-bb8f-1900f94074e8-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:07.915036 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.914910 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f9920297-9a26-49bd-bb8f-1900f94074e8-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:07.915036 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.915010 5274 scope.go:122] "RemoveContainer" containerID="448e85686ef9d9ac38feb4d6310dc88bd700706e235512a056967dadc4bb6473" Oct 31 00:08:07.926014 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:08:07.925907 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"448e85686ef9d9ac38feb4d6310dc88bd700706e235512a056967dadc4bb6473\": container with ID starting with 448e85686ef9d9ac38feb4d6310dc88bd700706e235512a056967dadc4bb6473 not found: ID does not exist" containerID="448e85686ef9d9ac38feb4d6310dc88bd700706e235512a056967dadc4bb6473" Oct 31 00:08:07.926014 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.925994 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"448e85686ef9d9ac38feb4d6310dc88bd700706e235512a056967dadc4bb6473"} err="failed to get container status \"448e85686ef9d9ac38feb4d6310dc88bd700706e235512a056967dadc4bb6473\": rpc error: code = NotFound desc = could not find container \"448e85686ef9d9ac38feb4d6310dc88bd700706e235512a056967dadc4bb6473\": container with ID starting with 448e85686ef9d9ac38feb4d6310dc88bd700706e235512a056967dadc4bb6473 not found: ID does not exist" Oct 31 00:08:07.926628 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.926020 5274 scope.go:122] "RemoveContainer" containerID="b063bf2f1aa10d5003a6652b9fc76df68372541201b8a6e2d0252fa2bed46767" Oct 31 00:08:07.929535 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.928307 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9920297-9a26-49bd-bb8f-1900f94074e8-prometheus-data-science-monitoringstack-rulefiles-1" pod "f9920297-9a26-49bd-bb8f-1900f94074e8" (UID: "f9920297-9a26-49bd-bb8f-1900f94074e8"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:07.929535 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.929490 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9920297-9a26-49bd-bb8f-1900f94074e8-config-out" pod "f9920297-9a26-49bd-bb8f-1900f94074e8" (UID: "f9920297-9a26-49bd-bb8f-1900f94074e8"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:07.929902 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.929879 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-gvvck"] Oct 31 00:08:07.931591 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.931555 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-gvvck"] Oct 31 00:08:07.933617 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.933578 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-thanos-prometheus-http-client-file" pod "f9920297-9a26-49bd-bb8f-1900f94074e8" (UID: "f9920297-9a26-49bd-bb8f-1900f94074e8"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:07.936356 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.936332 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "f9920297-9a26-49bd-bb8f-1900f94074e8" (UID: "f9920297-9a26-49bd-bb8f-1900f94074e8"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:07.936495 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.936372 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-config" pod "f9920297-9a26-49bd-bb8f-1900f94074e8" (UID: "f9920297-9a26-49bd-bb8f-1900f94074e8"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:07.936844 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.936818 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9920297-9a26-49bd-bb8f-1900f94074e8-tls-assets" pod "f9920297-9a26-49bd-bb8f-1900f94074e8" (UID: "f9920297-9a26-49bd-bb8f-1900f94074e8"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:07.936932 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.936873 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "f9920297-9a26-49bd-bb8f-1900f94074e8" (UID: "f9920297-9a26-49bd-bb8f-1900f94074e8"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:07.937603 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.937580 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1" pod "f9920297-9a26-49bd-bb8f-1900f94074e8" (UID: "f9920297-9a26-49bd-bb8f-1900f94074e8"). InnerVolumeSpecName "pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:08:07.938366 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.938345 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9920297-9a26-49bd-bb8f-1900f94074e8-kube-api-access-j8vxf" pod "f9920297-9a26-49bd-bb8f-1900f94074e8" (UID: "f9920297-9a26-49bd-bb8f-1900f94074e8"). InnerVolumeSpecName "kube-api-access-j8vxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:07.938366 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.938348 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-secret-prometheus-web-tls-ca" pod "f9920297-9a26-49bd-bb8f-1900f94074e8" (UID: "f9920297-9a26-49bd-bb8f-1900f94074e8"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:07.938762 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.938747 5274 scope.go:122] "RemoveContainer" containerID="5e0c57752d1d7afa29ec624f12c2f45abce2e376eccb9f4976d2c62a879f5519" Oct 31 00:08:07.945430 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.945402 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-web-config" pod "f9920297-9a26-49bd-bb8f-1900f94074e8" (UID: "f9920297-9a26-49bd-bb8f-1900f94074e8"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:07.946836 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.946815 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nbzfw"] Oct 31 00:08:07.947671 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.947655 5274 scope.go:122] "RemoveContainer" containerID="9d38a5b791efb7a0922794fdac8bbde4d4385077a526b14f021892fc87d634a6" Oct 31 00:08:07.951232 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.951211 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-nbzfw"] Oct 31 00:08:07.956335 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.956320 5274 scope.go:122] "RemoveContainer" containerID="2fb428f880e65496e654d888ceea1168a1f6de2d4b18986ba8600ac51ce8b44e" Oct 31 00:08:07.964144 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.964124 5274 scope.go:122] "RemoveContainer" containerID="b063bf2f1aa10d5003a6652b9fc76df68372541201b8a6e2d0252fa2bed46767" Oct 31 00:08:07.964428 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:08:07.964409 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b063bf2f1aa10d5003a6652b9fc76df68372541201b8a6e2d0252fa2bed46767\": container with ID starting with b063bf2f1aa10d5003a6652b9fc76df68372541201b8a6e2d0252fa2bed46767 not found: ID does not exist" containerID="b063bf2f1aa10d5003a6652b9fc76df68372541201b8a6e2d0252fa2bed46767" Oct 31 00:08:07.964486 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.964437 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b063bf2f1aa10d5003a6652b9fc76df68372541201b8a6e2d0252fa2bed46767"} err="failed to get container status \"b063bf2f1aa10d5003a6652b9fc76df68372541201b8a6e2d0252fa2bed46767\": rpc error: code = NotFound desc = could not find container \"b063bf2f1aa10d5003a6652b9fc76df68372541201b8a6e2d0252fa2bed46767\": container with ID starting with b063bf2f1aa10d5003a6652b9fc76df68372541201b8a6e2d0252fa2bed46767 not found: ID does not exist" Oct 31 00:08:07.964486 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.964457 5274 scope.go:122] "RemoveContainer" containerID="5e0c57752d1d7afa29ec624f12c2f45abce2e376eccb9f4976d2c62a879f5519" Oct 31 00:08:07.964712 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:08:07.964694 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e0c57752d1d7afa29ec624f12c2f45abce2e376eccb9f4976d2c62a879f5519\": container with ID starting with 5e0c57752d1d7afa29ec624f12c2f45abce2e376eccb9f4976d2c62a879f5519 not found: ID does not exist" containerID="5e0c57752d1d7afa29ec624f12c2f45abce2e376eccb9f4976d2c62a879f5519" Oct 31 00:08:07.964761 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.964718 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e0c57752d1d7afa29ec624f12c2f45abce2e376eccb9f4976d2c62a879f5519"} err="failed to get container status \"5e0c57752d1d7afa29ec624f12c2f45abce2e376eccb9f4976d2c62a879f5519\": rpc error: code = NotFound desc = could not find container \"5e0c57752d1d7afa29ec624f12c2f45abce2e376eccb9f4976d2c62a879f5519\": container with ID starting with 5e0c57752d1d7afa29ec624f12c2f45abce2e376eccb9f4976d2c62a879f5519 not found: ID does not exist" Oct 31 00:08:07.964761 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.964737 5274 scope.go:122] "RemoveContainer" containerID="9d38a5b791efb7a0922794fdac8bbde4d4385077a526b14f021892fc87d634a6" Oct 31 00:08:07.964991 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:08:07.964974 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d38a5b791efb7a0922794fdac8bbde4d4385077a526b14f021892fc87d634a6\": container with ID starting with 9d38a5b791efb7a0922794fdac8bbde4d4385077a526b14f021892fc87d634a6 not found: ID does not exist" containerID="9d38a5b791efb7a0922794fdac8bbde4d4385077a526b14f021892fc87d634a6" Oct 31 00:08:07.965065 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.964994 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d38a5b791efb7a0922794fdac8bbde4d4385077a526b14f021892fc87d634a6"} err="failed to get container status \"9d38a5b791efb7a0922794fdac8bbde4d4385077a526b14f021892fc87d634a6\": rpc error: code = NotFound desc = could not find container \"9d38a5b791efb7a0922794fdac8bbde4d4385077a526b14f021892fc87d634a6\": container with ID starting with 9d38a5b791efb7a0922794fdac8bbde4d4385077a526b14f021892fc87d634a6 not found: ID does not exist" Oct 31 00:08:07.965065 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.965006 5274 scope.go:122] "RemoveContainer" containerID="2fb428f880e65496e654d888ceea1168a1f6de2d4b18986ba8600ac51ce8b44e" Oct 31 00:08:07.965216 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:08:07.965199 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fb428f880e65496e654d888ceea1168a1f6de2d4b18986ba8600ac51ce8b44e\": container with ID starting with 2fb428f880e65496e654d888ceea1168a1f6de2d4b18986ba8600ac51ce8b44e not found: ID does not exist" containerID="2fb428f880e65496e654d888ceea1168a1f6de2d4b18986ba8600ac51ce8b44e" Oct 31 00:08:07.965256 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.965220 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fb428f880e65496e654d888ceea1168a1f6de2d4b18986ba8600ac51ce8b44e"} err="failed to get container status \"2fb428f880e65496e654d888ceea1168a1f6de2d4b18986ba8600ac51ce8b44e\": rpc error: code = NotFound desc = could not find container \"2fb428f880e65496e654d888ceea1168a1f6de2d4b18986ba8600ac51ce8b44e\": container with ID starting with 2fb428f880e65496e654d888ceea1168a1f6de2d4b18986ba8600ac51ce8b44e not found: ID does not exist" Oct 31 00:08:07.965256 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.965231 5274 scope.go:122] "RemoveContainer" containerID="b063bf2f1aa10d5003a6652b9fc76df68372541201b8a6e2d0252fa2bed46767" Oct 31 00:08:07.965445 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.965422 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b063bf2f1aa10d5003a6652b9fc76df68372541201b8a6e2d0252fa2bed46767"} err="failed to get container status \"b063bf2f1aa10d5003a6652b9fc76df68372541201b8a6e2d0252fa2bed46767\": rpc error: code = NotFound desc = could not find container \"b063bf2f1aa10d5003a6652b9fc76df68372541201b8a6e2d0252fa2bed46767\": container with ID starting with b063bf2f1aa10d5003a6652b9fc76df68372541201b8a6e2d0252fa2bed46767 not found: ID does not exist" Oct 31 00:08:07.965491 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.965446 5274 scope.go:122] "RemoveContainer" containerID="5e0c57752d1d7afa29ec624f12c2f45abce2e376eccb9f4976d2c62a879f5519" Oct 31 00:08:07.965726 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.965710 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e0c57752d1d7afa29ec624f12c2f45abce2e376eccb9f4976d2c62a879f5519"} err="failed to get container status \"5e0c57752d1d7afa29ec624f12c2f45abce2e376eccb9f4976d2c62a879f5519\": rpc error: code = NotFound desc = could not find container \"5e0c57752d1d7afa29ec624f12c2f45abce2e376eccb9f4976d2c62a879f5519\": container with ID starting with 5e0c57752d1d7afa29ec624f12c2f45abce2e376eccb9f4976d2c62a879f5519 not found: ID does not exist" Oct 31 00:08:07.965777 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.965726 5274 scope.go:122] "RemoveContainer" containerID="9d38a5b791efb7a0922794fdac8bbde4d4385077a526b14f021892fc87d634a6" Oct 31 00:08:07.966061 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.966042 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d38a5b791efb7a0922794fdac8bbde4d4385077a526b14f021892fc87d634a6"} err="failed to get container status \"9d38a5b791efb7a0922794fdac8bbde4d4385077a526b14f021892fc87d634a6\": rpc error: code = NotFound desc = could not find container \"9d38a5b791efb7a0922794fdac8bbde4d4385077a526b14f021892fc87d634a6\": container with ID starting with 9d38a5b791efb7a0922794fdac8bbde4d4385077a526b14f021892fc87d634a6 not found: ID does not exist" Oct 31 00:08:07.966061 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.966061 5274 scope.go:122] "RemoveContainer" containerID="2fb428f880e65496e654d888ceea1168a1f6de2d4b18986ba8600ac51ce8b44e" Oct 31 00:08:07.966294 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.966275 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fb428f880e65496e654d888ceea1168a1f6de2d4b18986ba8600ac51ce8b44e"} err="failed to get container status \"2fb428f880e65496e654d888ceea1168a1f6de2d4b18986ba8600ac51ce8b44e\": rpc error: code = NotFound desc = could not find container \"2fb428f880e65496e654d888ceea1168a1f6de2d4b18986ba8600ac51ce8b44e\": container with ID starting with 2fb428f880e65496e654d888ceea1168a1f6de2d4b18986ba8600ac51ce8b44e not found: ID does not exist" Oct 31 00:08:07.966346 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.966294 5274 scope.go:122] "RemoveContainer" containerID="b063bf2f1aa10d5003a6652b9fc76df68372541201b8a6e2d0252fa2bed46767" Oct 31 00:08:07.966515 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.966496 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b063bf2f1aa10d5003a6652b9fc76df68372541201b8a6e2d0252fa2bed46767"} err="failed to get container status \"b063bf2f1aa10d5003a6652b9fc76df68372541201b8a6e2d0252fa2bed46767\": rpc error: code = NotFound desc = could not find container \"b063bf2f1aa10d5003a6652b9fc76df68372541201b8a6e2d0252fa2bed46767\": container with ID starting with b063bf2f1aa10d5003a6652b9fc76df68372541201b8a6e2d0252fa2bed46767 not found: ID does not exist" Oct 31 00:08:07.966515 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.966515 5274 scope.go:122] "RemoveContainer" containerID="5e0c57752d1d7afa29ec624f12c2f45abce2e376eccb9f4976d2c62a879f5519" Oct 31 00:08:07.966738 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.966718 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e0c57752d1d7afa29ec624f12c2f45abce2e376eccb9f4976d2c62a879f5519"} err="failed to get container status \"5e0c57752d1d7afa29ec624f12c2f45abce2e376eccb9f4976d2c62a879f5519\": rpc error: code = NotFound desc = could not find container \"5e0c57752d1d7afa29ec624f12c2f45abce2e376eccb9f4976d2c62a879f5519\": container with ID starting with 5e0c57752d1d7afa29ec624f12c2f45abce2e376eccb9f4976d2c62a879f5519 not found: ID does not exist" Oct 31 00:08:07.966738 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.966738 5274 scope.go:122] "RemoveContainer" containerID="9d38a5b791efb7a0922794fdac8bbde4d4385077a526b14f021892fc87d634a6" Oct 31 00:08:07.966933 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.966917 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d38a5b791efb7a0922794fdac8bbde4d4385077a526b14f021892fc87d634a6"} err="failed to get container status \"9d38a5b791efb7a0922794fdac8bbde4d4385077a526b14f021892fc87d634a6\": rpc error: code = NotFound desc = could not find container \"9d38a5b791efb7a0922794fdac8bbde4d4385077a526b14f021892fc87d634a6\": container with ID starting with 9d38a5b791efb7a0922794fdac8bbde4d4385077a526b14f021892fc87d634a6 not found: ID does not exist" Oct 31 00:08:07.966933 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.966933 5274 scope.go:122] "RemoveContainer" containerID="2fb428f880e65496e654d888ceea1168a1f6de2d4b18986ba8600ac51ce8b44e" Oct 31 00:08:07.967180 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:07.967147 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fb428f880e65496e654d888ceea1168a1f6de2d4b18986ba8600ac51ce8b44e"} err="failed to get container status \"2fb428f880e65496e654d888ceea1168a1f6de2d4b18986ba8600ac51ce8b44e\": rpc error: code = NotFound desc = could not find container \"2fb428f880e65496e654d888ceea1168a1f6de2d4b18986ba8600ac51ce8b44e\": container with ID starting with 2fb428f880e65496e654d888ceea1168a1f6de2d4b18986ba8600ac51ce8b44e not found: ID does not exist" Oct 31 00:08:08.015615 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.015592 5274 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.015615 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.015611 5274 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-web-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.015725 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.015640 5274 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" " Oct 31 00:08:08.015725 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.015650 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8vxf\" (UniqueName: \"kubernetes.io/projected/f9920297-9a26-49bd-bb8f-1900f94074e8-kube-api-access-j8vxf\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.015725 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.015659 5274 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9920297-9a26-49bd-bb8f-1900f94074e8-tls-assets\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.015725 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.015667 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.015725 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.015678 5274 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.015725 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.015687 5274 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.015725 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.015698 5274 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/f9920297-9a26-49bd-bb8f-1900f94074e8-secret-prometheus-web-tls-ca\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.015725 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.015706 5274 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9920297-9a26-49bd-bb8f-1900f94074e8-config-out\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.015725 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.015716 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f9920297-9a26-49bd-bb8f-1900f94074e8-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.029850 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.029835 5274 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1") on node "ip-10-0-129-70.us-east-2.compute.internal" Oct 31 00:08:08.116234 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.116205 5274 reconciler_common.go:299] "Volume detached for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"csi-51e8027f163b5f98040faf3f8c30848dd44a7a90ca2bd0af2446ea6398ed018d\"" Oct 31 00:08:08.261171 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.260534 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:08:08.266021 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.265996 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:08:08.364792 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.364772 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:08:08.519223 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.519145 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a99ee414-acb3-4091-9e6d-825a4c76a2e4-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/a99ee414-acb3-4091-9e6d-825a4c76a2e4-alertmanager-data-science-monitoringstack-db\") pod \"a99ee414-acb3-4091-9e6d-825a4c76a2e4\" (UID: \"a99ee414-acb3-4091-9e6d-825a4c76a2e4\") " Oct 31 00:08:08.519223 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.519208 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a99ee414-acb3-4091-9e6d-825a4c76a2e4-config-out\" (UniqueName: \"kubernetes.io/empty-dir/a99ee414-acb3-4091-9e6d-825a4c76a2e4-config-out\") pod \"a99ee414-acb3-4091-9e6d-825a4c76a2e4\" (UID: \"a99ee414-acb3-4091-9e6d-825a4c76a2e4\") " Oct 31 00:08:08.519500 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.519232 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a99ee414-acb3-4091-9e6d-825a4c76a2e4-tls-assets\" (UniqueName: \"kubernetes.io/projected/a99ee414-acb3-4091-9e6d-825a4c76a2e4-tls-assets\") pod \"a99ee414-acb3-4091-9e6d-825a4c76a2e4\" (UID: \"a99ee414-acb3-4091-9e6d-825a4c76a2e4\") " Oct 31 00:08:08.519500 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.519271 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a99ee414-acb3-4091-9e6d-825a4c76a2e4-config-volume\" (UniqueName: \"kubernetes.io/secret/a99ee414-acb3-4091-9e6d-825a4c76a2e4-config-volume\") pod \"a99ee414-acb3-4091-9e6d-825a4c76a2e4\" (UID: \"a99ee414-acb3-4091-9e6d-825a4c76a2e4\") " Oct 31 00:08:08.519500 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.519287 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a99ee414-acb3-4091-9e6d-825a4c76a2e4-web-config\" (UniqueName: \"kubernetes.io/secret/a99ee414-acb3-4091-9e6d-825a4c76a2e4-web-config\") pod \"a99ee414-acb3-4091-9e6d-825a4c76a2e4\" (UID: \"a99ee414-acb3-4091-9e6d-825a4c76a2e4\") " Oct 31 00:08:08.519500 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.519327 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a99ee414-acb3-4091-9e6d-825a4c76a2e4-kube-api-access-mfc5z\" (UniqueName: \"kubernetes.io/projected/a99ee414-acb3-4091-9e6d-825a4c76a2e4-kube-api-access-mfc5z\") pod \"a99ee414-acb3-4091-9e6d-825a4c76a2e4\" (UID: \"a99ee414-acb3-4091-9e6d-825a4c76a2e4\") " Oct 31 00:08:08.519500 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.519355 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a99ee414-acb3-4091-9e6d-825a4c76a2e4-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a99ee414-acb3-4091-9e6d-825a4c76a2e4-cluster-tls-config\") pod \"a99ee414-acb3-4091-9e6d-825a4c76a2e4\" (UID: \"a99ee414-acb3-4091-9e6d-825a4c76a2e4\") " Oct 31 00:08:08.519746 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.519514 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a99ee414-acb3-4091-9e6d-825a4c76a2e4-alertmanager-data-science-monitoringstack-db" pod "a99ee414-acb3-4091-9e6d-825a4c76a2e4" (UID: "a99ee414-acb3-4091-9e6d-825a4c76a2e4"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:08.521435 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.521409 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a99ee414-acb3-4091-9e6d-825a4c76a2e4-kube-api-access-mfc5z" pod "a99ee414-acb3-4091-9e6d-825a4c76a2e4" (UID: "a99ee414-acb3-4091-9e6d-825a4c76a2e4"). InnerVolumeSpecName "kube-api-access-mfc5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:08.521802 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.521772 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a99ee414-acb3-4091-9e6d-825a4c76a2e4-config-out" pod "a99ee414-acb3-4091-9e6d-825a4c76a2e4" (UID: "a99ee414-acb3-4091-9e6d-825a4c76a2e4"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:08.521885 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.521823 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a99ee414-acb3-4091-9e6d-825a4c76a2e4-tls-assets" pod "a99ee414-acb3-4091-9e6d-825a4c76a2e4" (UID: "a99ee414-acb3-4091-9e6d-825a4c76a2e4"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:08.521885 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.521861 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a99ee414-acb3-4091-9e6d-825a4c76a2e4-config-volume" pod "a99ee414-acb3-4091-9e6d-825a4c76a2e4" (UID: "a99ee414-acb3-4091-9e6d-825a4c76a2e4"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:08.525463 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.525443 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a99ee414-acb3-4091-9e6d-825a4c76a2e4-cluster-tls-config" pod "a99ee414-acb3-4091-9e6d-825a4c76a2e4" (UID: "a99ee414-acb3-4091-9e6d-825a4c76a2e4"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:08.532571 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.532549 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a99ee414-acb3-4091-9e6d-825a4c76a2e4-web-config" pod "a99ee414-acb3-4091-9e6d-825a4c76a2e4" (UID: "a99ee414-acb3-4091-9e6d-825a4c76a2e4"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:08.620634 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.620612 5274 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a99ee414-acb3-4091-9e6d-825a4c76a2e4-config-volume\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.620634 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.620634 5274 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a99ee414-acb3-4091-9e6d-825a4c76a2e4-web-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.620750 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.620645 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfc5z\" (UniqueName: \"kubernetes.io/projected/a99ee414-acb3-4091-9e6d-825a4c76a2e4-kube-api-access-mfc5z\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.620750 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.620654 5274 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a99ee414-acb3-4091-9e6d-825a4c76a2e4-cluster-tls-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.620750 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.620664 5274 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/a99ee414-acb3-4091-9e6d-825a4c76a2e4-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.620750 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.620673 5274 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a99ee414-acb3-4091-9e6d-825a4c76a2e4-config-out\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.620750 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.620681 5274 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a99ee414-acb3-4091-9e6d-825a4c76a2e4-tls-assets\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.910996 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.910945 5274 generic.go:356] "Generic (PLEG): container finished" podID="a99ee414-acb3-4091-9e6d-825a4c76a2e4" containerID="4a788c4f910fa7e357a3e1c5e36e87cb0b34c8cbd606f2a1a16dace07620ddce" exitCode=0 Oct 31 00:08:08.911161 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.911010 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"a99ee414-acb3-4091-9e6d-825a4c76a2e4","Type":"ContainerDied","Data":"4a788c4f910fa7e357a3e1c5e36e87cb0b34c8cbd606f2a1a16dace07620ddce"} Oct 31 00:08:08.911161 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.911041 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:08:08.911161 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.911056 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"a99ee414-acb3-4091-9e6d-825a4c76a2e4","Type":"ContainerDied","Data":"17d8754a73bd98a726da76d0bfa2adac4658aeda608d38503bcebb0acb537229"} Oct 31 00:08:08.911161 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.911084 5274 scope.go:122] "RemoveContainer" containerID="e312b7a1fabcb037d9ca35380ecec359f22af00646bbbd13ac0eb5cdef666bff" Oct 31 00:08:08.920017 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.920001 5274 scope.go:122] "RemoveContainer" containerID="4a788c4f910fa7e357a3e1c5e36e87cb0b34c8cbd606f2a1a16dace07620ddce" Oct 31 00:08:08.927808 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.927787 5274 scope.go:122] "RemoveContainer" containerID="1c68e1e44c8cc260672f41b0619ceecb24035b6ccec494b8a4c3d8dc21b22df6" Oct 31 00:08:08.934997 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.934974 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:08:08.935414 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.935400 5274 scope.go:122] "RemoveContainer" containerID="e312b7a1fabcb037d9ca35380ecec359f22af00646bbbd13ac0eb5cdef666bff" Oct 31 00:08:08.935665 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:08:08.935644 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e312b7a1fabcb037d9ca35380ecec359f22af00646bbbd13ac0eb5cdef666bff\": container with ID starting with e312b7a1fabcb037d9ca35380ecec359f22af00646bbbd13ac0eb5cdef666bff not found: ID does not exist" containerID="e312b7a1fabcb037d9ca35380ecec359f22af00646bbbd13ac0eb5cdef666bff" Oct 31 00:08:08.935733 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.935677 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e312b7a1fabcb037d9ca35380ecec359f22af00646bbbd13ac0eb5cdef666bff"} err="failed to get container status \"e312b7a1fabcb037d9ca35380ecec359f22af00646bbbd13ac0eb5cdef666bff\": rpc error: code = NotFound desc = could not find container \"e312b7a1fabcb037d9ca35380ecec359f22af00646bbbd13ac0eb5cdef666bff\": container with ID starting with e312b7a1fabcb037d9ca35380ecec359f22af00646bbbd13ac0eb5cdef666bff not found: ID does not exist" Oct 31 00:08:08.935733 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.935700 5274 scope.go:122] "RemoveContainer" containerID="4a788c4f910fa7e357a3e1c5e36e87cb0b34c8cbd606f2a1a16dace07620ddce" Oct 31 00:08:08.935937 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:08:08.935919 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a788c4f910fa7e357a3e1c5e36e87cb0b34c8cbd606f2a1a16dace07620ddce\": container with ID starting with 4a788c4f910fa7e357a3e1c5e36e87cb0b34c8cbd606f2a1a16dace07620ddce not found: ID does not exist" containerID="4a788c4f910fa7e357a3e1c5e36e87cb0b34c8cbd606f2a1a16dace07620ddce" Oct 31 00:08:08.936154 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.935946 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a788c4f910fa7e357a3e1c5e36e87cb0b34c8cbd606f2a1a16dace07620ddce"} err="failed to get container status \"4a788c4f910fa7e357a3e1c5e36e87cb0b34c8cbd606f2a1a16dace07620ddce\": rpc error: code = NotFound desc = could not find container \"4a788c4f910fa7e357a3e1c5e36e87cb0b34c8cbd606f2a1a16dace07620ddce\": container with ID starting with 4a788c4f910fa7e357a3e1c5e36e87cb0b34c8cbd606f2a1a16dace07620ddce not found: ID does not exist" Oct 31 00:08:08.936211 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.936157 5274 scope.go:122] "RemoveContainer" containerID="1c68e1e44c8cc260672f41b0619ceecb24035b6ccec494b8a4c3d8dc21b22df6" Oct 31 00:08:08.936405 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:08:08.936390 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c68e1e44c8cc260672f41b0619ceecb24035b6ccec494b8a4c3d8dc21b22df6\": container with ID starting with 1c68e1e44c8cc260672f41b0619ceecb24035b6ccec494b8a4c3d8dc21b22df6 not found: ID does not exist" containerID="1c68e1e44c8cc260672f41b0619ceecb24035b6ccec494b8a4c3d8dc21b22df6" Oct 31 00:08:08.936441 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.936408 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c68e1e44c8cc260672f41b0619ceecb24035b6ccec494b8a4c3d8dc21b22df6"} err="failed to get container status \"1c68e1e44c8cc260672f41b0619ceecb24035b6ccec494b8a4c3d8dc21b22df6\": rpc error: code = NotFound desc = could not find container \"1c68e1e44c8cc260672f41b0619ceecb24035b6ccec494b8a4c3d8dc21b22df6\": container with ID starting with 1c68e1e44c8cc260672f41b0619ceecb24035b6ccec494b8a4c3d8dc21b22df6 not found: ID does not exist" Oct 31 00:08:08.941686 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:08.941666 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:08:09.481237 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:09.481174 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a99ee414-acb3-4091-9e6d-825a4c76a2e4" path="/var/lib/kubelet/pods/a99ee414-acb3-4091-9e6d-825a4c76a2e4/volumes" Oct 31 00:08:09.481617 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:09.481604 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b041f16e-c90d-4083-b5ab-5a52412e739a" path="/var/lib/kubelet/pods/b041f16e-c90d-4083-b5ab-5a52412e739a/volumes" Oct 31 00:08:09.481939 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:09.481928 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec56f3ec-c11e-4358-8707-6723210bbebc" path="/var/lib/kubelet/pods/ec56f3ec-c11e-4358-8707-6723210bbebc/volumes" Oct 31 00:08:09.482689 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:09.482673 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9920297-9a26-49bd-bb8f-1900f94074e8" path="/var/lib/kubelet/pods/f9920297-9a26-49bd-bb8f-1900f94074e8/volumes" Oct 31 00:08:10.015100 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.015063 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-vzbjf"] Oct 31 00:08:10.015462 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.015445 5274 cpu_manager.go:414] "containerMap: removing container" podUID="ec56f3ec-c11e-4358-8707-6723210bbebc" containerName="kube-rbac-proxy" Oct 31 00:08:10.015553 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.015463 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec56f3ec-c11e-4358-8707-6723210bbebc" containerName="kube-rbac-proxy" Oct 31 00:08:10.015553 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.015483 5274 cpu_manager.go:414] "containerMap: removing container" podUID="f9920297-9a26-49bd-bb8f-1900f94074e8" containerName="prometheus" Oct 31 00:08:10.015553 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.015491 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9920297-9a26-49bd-bb8f-1900f94074e8" containerName="prometheus" Oct 31 00:08:10.015553 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.015505 5274 cpu_manager.go:414] "containerMap: removing container" podUID="f9920297-9a26-49bd-bb8f-1900f94074e8" containerName="init-config-reloader" Oct 31 00:08:10.015553 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.015513 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9920297-9a26-49bd-bb8f-1900f94074e8" containerName="init-config-reloader" Oct 31 00:08:10.015553 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.015524 5274 cpu_manager.go:414] "containerMap: removing container" podUID="a99ee414-acb3-4091-9e6d-825a4c76a2e4" containerName="config-reloader" Oct 31 00:08:10.015553 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.015531 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a99ee414-acb3-4091-9e6d-825a4c76a2e4" containerName="config-reloader" Oct 31 00:08:10.015553 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.015554 5274 cpu_manager.go:414] "containerMap: removing container" podUID="a99ee414-acb3-4091-9e6d-825a4c76a2e4" containerName="init-config-reloader" Oct 31 00:08:10.015921 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.015561 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a99ee414-acb3-4091-9e6d-825a4c76a2e4" containerName="init-config-reloader" Oct 31 00:08:10.015921 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.015573 5274 cpu_manager.go:414] "containerMap: removing container" podUID="f9920297-9a26-49bd-bb8f-1900f94074e8" containerName="thanos-sidecar" Oct 31 00:08:10.015921 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.015580 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9920297-9a26-49bd-bb8f-1900f94074e8" containerName="thanos-sidecar" Oct 31 00:08:10.015921 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.015592 5274 cpu_manager.go:414] "containerMap: removing container" podUID="f9920297-9a26-49bd-bb8f-1900f94074e8" containerName="config-reloader" Oct 31 00:08:10.015921 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.015599 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9920297-9a26-49bd-bb8f-1900f94074e8" containerName="config-reloader" Oct 31 00:08:10.015921 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.015612 5274 cpu_manager.go:414] "containerMap: removing container" podUID="b041f16e-c90d-4083-b5ab-5a52412e739a" containerName="korrel8r" Oct 31 00:08:10.015921 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.015619 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b041f16e-c90d-4083-b5ab-5a52412e739a" containerName="korrel8r" Oct 31 00:08:10.015921 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.015669 5274 cpu_manager.go:414] "containerMap: removing container" podUID="a99ee414-acb3-4091-9e6d-825a4c76a2e4" containerName="alertmanager" Oct 31 00:08:10.015921 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.015679 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a99ee414-acb3-4091-9e6d-825a4c76a2e4" containerName="alertmanager" Oct 31 00:08:10.015921 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.015767 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="b041f16e-c90d-4083-b5ab-5a52412e739a" containerName="korrel8r" Oct 31 00:08:10.015921 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.015780 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="a99ee414-acb3-4091-9e6d-825a4c76a2e4" containerName="alertmanager" Oct 31 00:08:10.015921 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.015792 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec56f3ec-c11e-4358-8707-6723210bbebc" containerName="kube-rbac-proxy" Oct 31 00:08:10.015921 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.015802 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9920297-9a26-49bd-bb8f-1900f94074e8" containerName="prometheus" Oct 31 00:08:10.015921 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.015812 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="a99ee414-acb3-4091-9e6d-825a4c76a2e4" containerName="config-reloader" Oct 31 00:08:10.015921 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.015822 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9920297-9a26-49bd-bb8f-1900f94074e8" containerName="thanos-sidecar" Oct 31 00:08:10.015921 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.015830 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9920297-9a26-49bd-bb8f-1900f94074e8" containerName="config-reloader" Oct 31 00:08:10.020579 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.020560 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-vzbjf" Oct 31 00:08:10.026263 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.026236 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 31 00:08:10.027536 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.027514 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 31 00:08:10.027637 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.027587 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:08:10.027815 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.027793 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:08:10.027914 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.027890 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-xlrwb\"" Oct 31 00:08:10.028128 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.028109 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-vzbjf"] Oct 31 00:08:10.132065 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.132039 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/3c64e116-a1aa-4fdd-bd37-b7ee8771e404-tls\") pod \"korrel8r-6644769df5-vzbjf\" (UID: \"3c64e116-a1aa-4fdd-bd37-b7ee8771e404\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-vzbjf" Oct 31 00:08:10.132228 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.132072 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c6pn\" (UniqueName: \"kubernetes.io/projected/3c64e116-a1aa-4fdd-bd37-b7ee8771e404-kube-api-access-8c6pn\") pod \"korrel8r-6644769df5-vzbjf\" (UID: \"3c64e116-a1aa-4fdd-bd37-b7ee8771e404\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-vzbjf" Oct 31 00:08:10.132228 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.132191 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3c64e116-a1aa-4fdd-bd37-b7ee8771e404-config\") pod \"korrel8r-6644769df5-vzbjf\" (UID: \"3c64e116-a1aa-4fdd-bd37-b7ee8771e404\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-vzbjf" Oct 31 00:08:10.233598 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.233570 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3c64e116-a1aa-4fdd-bd37-b7ee8771e404-config\") pod \"korrel8r-6644769df5-vzbjf\" (UID: \"3c64e116-a1aa-4fdd-bd37-b7ee8771e404\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-vzbjf" Oct 31 00:08:10.233769 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.233613 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/3c64e116-a1aa-4fdd-bd37-b7ee8771e404-tls\") pod \"korrel8r-6644769df5-vzbjf\" (UID: \"3c64e116-a1aa-4fdd-bd37-b7ee8771e404\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-vzbjf" Oct 31 00:08:10.233769 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.233653 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c6pn\" (UniqueName: \"kubernetes.io/projected/3c64e116-a1aa-4fdd-bd37-b7ee8771e404-kube-api-access-8c6pn\") pod \"korrel8r-6644769df5-vzbjf\" (UID: \"3c64e116-a1aa-4fdd-bd37-b7ee8771e404\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-vzbjf" Oct 31 00:08:10.234745 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.234708 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3c64e116-a1aa-4fdd-bd37-b7ee8771e404-config\") pod \"korrel8r-6644769df5-vzbjf\" (UID: \"3c64e116-a1aa-4fdd-bd37-b7ee8771e404\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-vzbjf" Oct 31 00:08:10.237739 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.237711 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/3c64e116-a1aa-4fdd-bd37-b7ee8771e404-tls\") pod \"korrel8r-6644769df5-vzbjf\" (UID: \"3c64e116-a1aa-4fdd-bd37-b7ee8771e404\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-vzbjf" Oct 31 00:08:10.255968 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.255926 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c6pn\" (UniqueName: \"kubernetes.io/projected/3c64e116-a1aa-4fdd-bd37-b7ee8771e404-kube-api-access-8c6pn\") pod \"korrel8r-6644769df5-vzbjf\" (UID: \"3c64e116-a1aa-4fdd-bd37-b7ee8771e404\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-vzbjf" Oct 31 00:08:10.334517 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.334440 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-vzbjf" Oct 31 00:08:10.608501 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.608421 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-vzbjf"] Oct 31 00:08:10.611848 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:08:10.611820 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c64e116_a1aa_4fdd_bd37_b7ee8771e404.slice/crio-1b1308f0d554cfc75ecdcee691d7d9ab871a9b73cde09a634b52b37c600f2ea3 WatchSource:0}: Error finding container 1b1308f0d554cfc75ecdcee691d7d9ab871a9b73cde09a634b52b37c600f2ea3: Status 404 returned error can't find the container with id 1b1308f0d554cfc75ecdcee691d7d9ab871a9b73cde09a634b52b37c600f2ea3 Oct 31 00:08:10.922808 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.922779 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-vzbjf" event={"ID":"3c64e116-a1aa-4fdd-bd37-b7ee8771e404","Type":"ContainerStarted","Data":"3268415cefeb91c6974d8031b2573a4f4338f23ec3922276c2b7d5dbf6f7abdd"} Oct 31 00:08:10.922989 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.922825 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-vzbjf" event={"ID":"3c64e116-a1aa-4fdd-bd37-b7ee8771e404","Type":"ContainerStarted","Data":"1b1308f0d554cfc75ecdcee691d7d9ab871a9b73cde09a634b52b37c600f2ea3"} Oct 31 00:08:10.922989 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.922922 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-vzbjf" Oct 31 00:08:10.945449 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:10.945405 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-6644769df5-vzbjf" podStartSLOduration=1.9453935530000002 podStartE2EDuration="1.945393553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:08:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:08:10.944541674 +0000 UTC m=+527.581360389" watchObservedRunningTime="2026-10-31 00:08:10.945393553 +0000 UTC m=+527.582212268" Oct 31 00:08:11.010371 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:11.010344 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:08:11.014056 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:11.014036 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:08:11.028466 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:11.028439 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:08:11.060329 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:11.060307 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\"" Oct 31 00:08:11.060447 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:11.060326 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 31 00:08:11.060447 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:11.060409 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\"" Oct 31 00:08:11.060447 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:11.060416 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\"" Oct 31 00:08:11.067550 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:11.067532 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\"" Oct 31 00:08:11.067677 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:11.067663 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 31 00:08:11.067730 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:11.067665 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\"" Oct 31 00:08:11.067730 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:11.067723 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-ghv5s\"" Oct 31 00:08:11.141747 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:11.141722 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vpdm\" (UniqueName: \"kubernetes.io/projected/4401656b-3d34-4bf5-956b-3a0fc555064f-kube-api-access-9vpdm\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:08:11.141842 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:11.141771 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/4401656b-3d34-4bf5-956b-3a0fc555064f-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:08:11.141904 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:11.141847 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/4401656b-3d34-4bf5-956b-3a0fc555064f-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:08:11.142027 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:11.142006 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/4401656b-3d34-4bf5-956b-3a0fc555064f-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:08:11.142097 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:11.142051 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/4401656b-3d34-4bf5-956b-3a0fc555064f-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:08:11.142157 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:11.142141 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/4401656b-3d34-4bf5-956b-3a0fc555064f-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:08:11.142198 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:11.142185 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/4401656b-3d34-4bf5-956b-3a0fc555064f-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:08:11.142248 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:11.142208 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4401656b-3d34-4bf5-956b-3a0fc555064f-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:08:11.242921 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:11.242845 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/4401656b-3d34-4bf5-956b-3a0fc555064f-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:08:11.242921 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:11.242888 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4401656b-3d34-4bf5-956b-3a0fc555064f-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:08:11.243119 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:11.242930 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vpdm\" (UniqueName: \"kubernetes.io/projected/4401656b-3d34-4bf5-956b-3a0fc555064f-kube-api-access-9vpdm\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:08:11.243119 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:11.242989 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/4401656b-3d34-4bf5-956b-3a0fc555064f-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:08:11.243119 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:11.243018 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/4401656b-3d34-4bf5-956b-3a0fc555064f-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:08:11.243119 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:11.243075 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/4401656b-3d34-4bf5-956b-3a0fc555064f-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:08:11.243119 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:11.243107 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/4401656b-3d34-4bf5-956b-3a0fc555064f-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:08:11.243343 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:11.243143 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/4401656b-3d34-4bf5-956b-3a0fc555064f-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:08:11.243879 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:11.243852 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/4401656b-3d34-4bf5-956b-3a0fc555064f-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:08:11.244059 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:11.244036 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4401656b-3d34-4bf5-956b-3a0fc555064f-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:08:11.244123 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:11.244062 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/4401656b-3d34-4bf5-956b-3a0fc555064f-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:08:11.244201 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:11.244178 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/4401656b-3d34-4bf5-956b-3a0fc555064f-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:08:11.246003 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:11.245908 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/4401656b-3d34-4bf5-956b-3a0fc555064f-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:08:11.246222 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:11.246200 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/4401656b-3d34-4bf5-956b-3a0fc555064f-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:08:11.246287 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:11.246245 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/4401656b-3d34-4bf5-956b-3a0fc555064f-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:08:11.255769 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:11.255747 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vpdm\" (UniqueName: \"kubernetes.io/projected/4401656b-3d34-4bf5-956b-3a0fc555064f-kube-api-access-9vpdm\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:08:15.172998 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:15.172943 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:08:15.180242 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:15.180213 5274 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:08:15.273405 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:15.273378 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:08:15.273525 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:15.273484 5274 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:08:15.276758 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:15.276742 5274 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") DevicePath \"csi-9ce3c80b5d5deac9dfdb953db4fb017080b654a85db76a2937e67cd541d89057\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:08:15.366613 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:15.366583 5274 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/151619720cb39d40781bf258206fb0be746b77e5f55879e189e4cf0d6e46dfb0/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:08:15.382200 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:15.382180 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:08:15.530994 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:15.530916 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:08:15.812011 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:15.811929 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:08:15.815289 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:08:15.815261 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4401656b_3d34_4bf5_956b_3a0fc555064f.slice/crio-c7484c05dcac74026b3c19a2372b0518a5541f4a77648396e45c4808172ed031 WatchSource:0}: Error finding container c7484c05dcac74026b3c19a2372b0518a5541f4a77648396e45c4808172ed031: Status 404 returned error can't find the container with id c7484c05dcac74026b3c19a2372b0518a5541f4a77648396e45c4808172ed031 Oct 31 00:08:15.951010 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:15.950897 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"4401656b-3d34-4bf5-956b-3a0fc555064f","Type":"ContainerStarted","Data":"c52b2ef878da6b12060990377a6804c5ba4388f48c2403866b3c0dc4dee7a551"} Oct 31 00:08:15.951010 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:15.950984 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"4401656b-3d34-4bf5-956b-3a0fc555064f","Type":"ContainerStarted","Data":"11b8de1f7bc9ccce7c8998af77fe871bc23ded88d6a4b80bd4b08333c79f8bb2"} Oct 31 00:08:15.951010 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:15.951010 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"4401656b-3d34-4bf5-956b-3a0fc555064f","Type":"ContainerStarted","Data":"c7484c05dcac74026b3c19a2372b0518a5541f4a77648396e45c4808172ed031"} Oct 31 00:08:16.958419 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:16.958385 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"4401656b-3d34-4bf5-956b-3a0fc555064f","Type":"ContainerStarted","Data":"f88b316f81ed9a1f1ac59afa00e48e5080b566e1a48727b62305b778d8f6fbc3"} Oct 31 00:08:16.958864 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:16.958623 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:08:16.958864 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:16.958670 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:08:16.958864 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:16.958694 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:08:16.961152 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:16.961129 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:08:16.967632 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:16.967613 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:08:16.987740 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:16.987694 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=6.987681381 podStartE2EDuration="6.987681381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:08:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:08:16.985904255 +0000 UTC m=+533.622722970" watchObservedRunningTime="2026-10-31 00:08:16.987681381 +0000 UTC m=+533.624500096" Oct 31 00:08:20.899234 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:08:20.899133 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:08:21.932608 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:21.932574 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-vzbjf" Oct 31 00:08:38.970967 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:38.970867 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="4401656b-3d34-4bf5-956b-3a0fc555064f" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:08:48.971031 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:48.970983 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="4401656b-3d34-4bf5-956b-3a0fc555064f" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:08:50.898918 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:08:50.898881 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:08:58.970090 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:08:58.970054 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:00.187762 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:00.187729 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-vzbjf"] Oct 31 00:09:00.188201 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:00.187950 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-6644769df5-vzbjf" podUID="3c64e116-a1aa-4fdd-bd37-b7ee8771e404" containerName="korrel8r" containerID="cri-o://3268415cefeb91c6974d8031b2573a4f4338f23ec3922276c2b7d5dbf6f7abdd" gracePeriod=30 Oct 31 00:09:00.405472 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:00.405434 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:09:00.405744 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:00.405691 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="4401656b-3d34-4bf5-956b-3a0fc555064f" containerName="tempo" containerID="cri-o://11b8de1f7bc9ccce7c8998af77fe871bc23ded88d6a4b80bd4b08333c79f8bb2" gracePeriod=30 Oct 31 00:09:00.405940 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:00.405895 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="4401656b-3d34-4bf5-956b-3a0fc555064f" containerName="tempo-gateway" containerID="cri-o://c52b2ef878da6b12060990377a6804c5ba4388f48c2403866b3c0dc4dee7a551" gracePeriod=30 Oct 31 00:09:00.405940 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:00.405910 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="4401656b-3d34-4bf5-956b-3a0fc555064f" containerName="tempo-gateway-opa" containerID="cri-o://f88b316f81ed9a1f1ac59afa00e48e5080b566e1a48727b62305b778d8f6fbc3" gracePeriod=30 Oct 31 00:09:00.533511 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:00.533487 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-vzbjf" Oct 31 00:09:00.653397 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:00.653366 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c64e116-a1aa-4fdd-bd37-b7ee8771e404-tls\" (UniqueName: \"kubernetes.io/secret/3c64e116-a1aa-4fdd-bd37-b7ee8771e404-tls\") pod \"3c64e116-a1aa-4fdd-bd37-b7ee8771e404\" (UID: \"3c64e116-a1aa-4fdd-bd37-b7ee8771e404\") " Oct 31 00:09:00.653546 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:00.653462 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c64e116-a1aa-4fdd-bd37-b7ee8771e404-kube-api-access-8c6pn\" (UniqueName: \"kubernetes.io/projected/3c64e116-a1aa-4fdd-bd37-b7ee8771e404-kube-api-access-8c6pn\") pod \"3c64e116-a1aa-4fdd-bd37-b7ee8771e404\" (UID: \"3c64e116-a1aa-4fdd-bd37-b7ee8771e404\") " Oct 31 00:09:00.653546 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:00.653506 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c64e116-a1aa-4fdd-bd37-b7ee8771e404-config\" (UniqueName: \"kubernetes.io/configmap/3c64e116-a1aa-4fdd-bd37-b7ee8771e404-config\") pod \"3c64e116-a1aa-4fdd-bd37-b7ee8771e404\" (UID: \"3c64e116-a1aa-4fdd-bd37-b7ee8771e404\") " Oct 31 00:09:00.653884 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:00.653857 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c64e116-a1aa-4fdd-bd37-b7ee8771e404-config" pod "3c64e116-a1aa-4fdd-bd37-b7ee8771e404" (UID: "3c64e116-a1aa-4fdd-bd37-b7ee8771e404"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:09:00.655434 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:00.655405 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c64e116-a1aa-4fdd-bd37-b7ee8771e404-tls" pod "3c64e116-a1aa-4fdd-bd37-b7ee8771e404" (UID: "3c64e116-a1aa-4fdd-bd37-b7ee8771e404"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:09:00.655434 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:00.655412 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c64e116-a1aa-4fdd-bd37-b7ee8771e404-kube-api-access-8c6pn" pod "3c64e116-a1aa-4fdd-bd37-b7ee8771e404" (UID: "3c64e116-a1aa-4fdd-bd37-b7ee8771e404"). InnerVolumeSpecName "kube-api-access-8c6pn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:09:00.755092 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:00.754987 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c6pn\" (UniqueName: \"kubernetes.io/projected/3c64e116-a1aa-4fdd-bd37-b7ee8771e404-kube-api-access-8c6pn\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:00.755092 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:00.755033 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3c64e116-a1aa-4fdd-bd37-b7ee8771e404-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:00.755092 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:00.755046 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/3c64e116-a1aa-4fdd-bd37-b7ee8771e404-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:01.177466 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.177435 5274 generic.go:356] "Generic (PLEG): container finished" podID="3c64e116-a1aa-4fdd-bd37-b7ee8771e404" containerID="3268415cefeb91c6974d8031b2573a4f4338f23ec3922276c2b7d5dbf6f7abdd" exitCode=0 Oct 31 00:09:01.177647 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.177485 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-vzbjf" Oct 31 00:09:01.177647 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.177521 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-vzbjf" event={"ID":"3c64e116-a1aa-4fdd-bd37-b7ee8771e404","Type":"ContainerDied","Data":"3268415cefeb91c6974d8031b2573a4f4338f23ec3922276c2b7d5dbf6f7abdd"} Oct 31 00:09:01.177647 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.177582 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-vzbjf" event={"ID":"3c64e116-a1aa-4fdd-bd37-b7ee8771e404","Type":"ContainerDied","Data":"1b1308f0d554cfc75ecdcee691d7d9ab871a9b73cde09a634b52b37c600f2ea3"} Oct 31 00:09:01.177647 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.177607 5274 scope.go:122] "RemoveContainer" containerID="3268415cefeb91c6974d8031b2573a4f4338f23ec3922276c2b7d5dbf6f7abdd" Oct 31 00:09:01.179779 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.179756 5274 generic.go:356] "Generic (PLEG): container finished" podID="4401656b-3d34-4bf5-956b-3a0fc555064f" containerID="f88b316f81ed9a1f1ac59afa00e48e5080b566e1a48727b62305b778d8f6fbc3" exitCode=0 Oct 31 00:09:01.179779 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.179778 5274 generic.go:356] "Generic (PLEG): container finished" podID="4401656b-3d34-4bf5-956b-3a0fc555064f" containerID="11b8de1f7bc9ccce7c8998af77fe871bc23ded88d6a4b80bd4b08333c79f8bb2" exitCode=0 Oct 31 00:09:01.179907 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.179796 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"4401656b-3d34-4bf5-956b-3a0fc555064f","Type":"ContainerDied","Data":"f88b316f81ed9a1f1ac59afa00e48e5080b566e1a48727b62305b778d8f6fbc3"} Oct 31 00:09:01.179907 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.179837 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"4401656b-3d34-4bf5-956b-3a0fc555064f","Type":"ContainerDied","Data":"11b8de1f7bc9ccce7c8998af77fe871bc23ded88d6a4b80bd4b08333c79f8bb2"} Oct 31 00:09:01.186474 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.186456 5274 scope.go:122] "RemoveContainer" containerID="3268415cefeb91c6974d8031b2573a4f4338f23ec3922276c2b7d5dbf6f7abdd" Oct 31 00:09:01.186727 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:09:01.186706 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3268415cefeb91c6974d8031b2573a4f4338f23ec3922276c2b7d5dbf6f7abdd\": container with ID starting with 3268415cefeb91c6974d8031b2573a4f4338f23ec3922276c2b7d5dbf6f7abdd not found: ID does not exist" containerID="3268415cefeb91c6974d8031b2573a4f4338f23ec3922276c2b7d5dbf6f7abdd" Oct 31 00:09:01.186768 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.186735 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3268415cefeb91c6974d8031b2573a4f4338f23ec3922276c2b7d5dbf6f7abdd"} err="failed to get container status \"3268415cefeb91c6974d8031b2573a4f4338f23ec3922276c2b7d5dbf6f7abdd\": rpc error: code = NotFound desc = could not find container \"3268415cefeb91c6974d8031b2573a4f4338f23ec3922276c2b7d5dbf6f7abdd\": container with ID starting with 3268415cefeb91c6974d8031b2573a4f4338f23ec3922276c2b7d5dbf6f7abdd not found: ID does not exist" Oct 31 00:09:01.200490 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.200470 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-vzbjf"] Oct 31 00:09:01.205767 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.205749 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-vzbjf"] Oct 31 00:09:01.480774 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.480704 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c64e116-a1aa-4fdd-bd37-b7ee8771e404" path="/var/lib/kubelet/pods/3c64e116-a1aa-4fdd-bd37-b7ee8771e404/volumes" Oct 31 00:09:01.659997 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.659974 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:01.762430 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.762356 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4401656b-3d34-4bf5-956b-3a0fc555064f-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/4401656b-3d34-4bf5-956b-3a0fc555064f-tempo-data-science-tempomonolithic-mtls\") pod \"4401656b-3d34-4bf5-956b-3a0fc555064f\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") " Oct 31 00:09:01.762430 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.762415 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4401656b-3d34-4bf5-956b-3a0fc555064f-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/4401656b-3d34-4bf5-956b-3a0fc555064f-tempo-conf\") pod \"4401656b-3d34-4bf5-956b-3a0fc555064f\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") " Oct 31 00:09:01.762657 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.762456 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4401656b-3d34-4bf5-956b-3a0fc555064f-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/4401656b-3d34-4bf5-956b-3a0fc555064f-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"4401656b-3d34-4bf5-956b-3a0fc555064f\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") " Oct 31 00:09:01.762657 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.762480 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4401656b-3d34-4bf5-956b-3a0fc555064f-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4401656b-3d34-4bf5-956b-3a0fc555064f-tempo-data-science-tempomonolithic-ca-bundle\") pod \"4401656b-3d34-4bf5-956b-3a0fc555064f\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") " Oct 31 00:09:01.762657 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.762523 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4401656b-3d34-4bf5-956b-3a0fc555064f-kube-api-access-9vpdm\" (UniqueName: \"kubernetes.io/projected/4401656b-3d34-4bf5-956b-3a0fc555064f-kube-api-access-9vpdm\") pod \"4401656b-3d34-4bf5-956b-3a0fc555064f\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") " Oct 31 00:09:01.762657 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.762550 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4401656b-3d34-4bf5-956b-3a0fc555064f-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/4401656b-3d34-4bf5-956b-3a0fc555064f-gateway-rbac\") pod \"4401656b-3d34-4bf5-956b-3a0fc555064f\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") " Oct 31 00:09:01.762657 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.762589 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4401656b-3d34-4bf5-956b-3a0fc555064f-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/4401656b-3d34-4bf5-956b-3a0fc555064f-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"4401656b-3d34-4bf5-956b-3a0fc555064f\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") " Oct 31 00:09:01.762657 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.762621 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4401656b-3d34-4bf5-956b-3a0fc555064f-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/4401656b-3d34-4bf5-956b-3a0fc555064f-gateway-tenants\") pod \"4401656b-3d34-4bf5-956b-3a0fc555064f\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") " Oct 31 00:09:01.762949 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.762712 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"4401656b-3d34-4bf5-956b-3a0fc555064f\" (UID: \"4401656b-3d34-4bf5-956b-3a0fc555064f\") " Oct 31 00:09:01.763050 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.763027 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4401656b-3d34-4bf5-956b-3a0fc555064f-tempo-conf" pod "4401656b-3d34-4bf5-956b-3a0fc555064f" (UID: "4401656b-3d34-4bf5-956b-3a0fc555064f"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:09:01.763189 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.763152 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4401656b-3d34-4bf5-956b-3a0fc555064f-tempo-data-science-tempomonolithic-serving-cabundle" pod "4401656b-3d34-4bf5-956b-3a0fc555064f" (UID: "4401656b-3d34-4bf5-956b-3a0fc555064f"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:09:01.763189 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.763150 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4401656b-3d34-4bf5-956b-3a0fc555064f-gateway-rbac" pod "4401656b-3d34-4bf5-956b-3a0fc555064f" (UID: "4401656b-3d34-4bf5-956b-3a0fc555064f"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:09:01.763289 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.763158 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4401656b-3d34-4bf5-956b-3a0fc555064f-tempo-data-science-tempomonolithic-ca-bundle" pod "4401656b-3d34-4bf5-956b-3a0fc555064f" (UID: "4401656b-3d34-4bf5-956b-3a0fc555064f"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:09:01.765084 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.765057 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4401656b-3d34-4bf5-956b-3a0fc555064f-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "4401656b-3d34-4bf5-956b-3a0fc555064f" (UID: "4401656b-3d34-4bf5-956b-3a0fc555064f"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:09:01.765185 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.765141 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4401656b-3d34-4bf5-956b-3a0fc555064f-kube-api-access-9vpdm" pod "4401656b-3d34-4bf5-956b-3a0fc555064f" (UID: "4401656b-3d34-4bf5-956b-3a0fc555064f"). InnerVolumeSpecName "kube-api-access-9vpdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:09:01.765258 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.765230 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4401656b-3d34-4bf5-956b-3a0fc555064f-tempo-data-science-tempomonolithic-mtls" pod "4401656b-3d34-4bf5-956b-3a0fc555064f" (UID: "4401656b-3d34-4bf5-956b-3a0fc555064f"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:09:01.765427 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.765403 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4401656b-3d34-4bf5-956b-3a0fc555064f-gateway-tenants" pod "4401656b-3d34-4bf5-956b-3a0fc555064f" (UID: "4401656b-3d34-4bf5-956b-3a0fc555064f"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:09:01.765821 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.765802 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062" pod "4401656b-3d34-4bf5-956b-3a0fc555064f" (UID: "4401656b-3d34-4bf5-956b-3a0fc555064f"). InnerVolumeSpecName "pvc-4491ac70-4597-43a7-a792-f0813314e433". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:09:01.863656 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.863629 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/4401656b-3d34-4bf5-956b-3a0fc555064f-tempo-conf\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:01.863656 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.863653 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/4401656b-3d34-4bf5-956b-3a0fc555064f-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:01.863833 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.863667 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4401656b-3d34-4bf5-956b-3a0fc555064f-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:01.863833 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.863682 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vpdm\" (UniqueName: \"kubernetes.io/projected/4401656b-3d34-4bf5-956b-3a0fc555064f-kube-api-access-9vpdm\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:01.863833 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.863691 5274 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/4401656b-3d34-4bf5-956b-3a0fc555064f-gateway-rbac\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:01.863833 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.863701 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/4401656b-3d34-4bf5-956b-3a0fc555064f-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:01.863833 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.863710 5274 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/4401656b-3d34-4bf5-956b-3a0fc555064f-gateway-tenants\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:01.863833 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.863737 5274 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" " Oct 31 00:09:01.863833 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.863751 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/4401656b-3d34-4bf5-956b-3a0fc555064f-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:01.872346 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.872328 5274 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-4491ac70-4597-43a7-a792-f0813314e433" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062") on node "ip-10-0-129-70.us-east-2.compute.internal" Oct 31 00:09:01.964924 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:01.964893 5274 reconciler_common.go:299] "Volume detached for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"csi-9ce3c80b5d5deac9dfdb953db4fb017080b654a85db76a2937e67cd541d89057\"" Oct 31 00:09:02.187296 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:02.187258 5274 generic.go:356] "Generic (PLEG): container finished" podID="4401656b-3d34-4bf5-956b-3a0fc555064f" containerID="c52b2ef878da6b12060990377a6804c5ba4388f48c2403866b3c0dc4dee7a551" exitCode=0 Oct 31 00:09:02.187453 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:02.187342 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:02.187453 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:02.187346 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"4401656b-3d34-4bf5-956b-3a0fc555064f","Type":"ContainerDied","Data":"c52b2ef878da6b12060990377a6804c5ba4388f48c2403866b3c0dc4dee7a551"} Oct 31 00:09:02.187453 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:02.187428 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"4401656b-3d34-4bf5-956b-3a0fc555064f","Type":"ContainerDied","Data":"c7484c05dcac74026b3c19a2372b0518a5541f4a77648396e45c4808172ed031"} Oct 31 00:09:02.187453 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:02.187453 5274 scope.go:122] "RemoveContainer" containerID="f88b316f81ed9a1f1ac59afa00e48e5080b566e1a48727b62305b778d8f6fbc3" Oct 31 00:09:02.195880 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:02.195864 5274 scope.go:122] "RemoveContainer" containerID="c52b2ef878da6b12060990377a6804c5ba4388f48c2403866b3c0dc4dee7a551" Oct 31 00:09:02.204915 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:02.204900 5274 scope.go:122] "RemoveContainer" containerID="11b8de1f7bc9ccce7c8998af77fe871bc23ded88d6a4b80bd4b08333c79f8bb2" Oct 31 00:09:02.212595 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:02.212577 5274 scope.go:122] "RemoveContainer" containerID="f88b316f81ed9a1f1ac59afa00e48e5080b566e1a48727b62305b778d8f6fbc3" Oct 31 00:09:02.212846 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:09:02.212827 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f88b316f81ed9a1f1ac59afa00e48e5080b566e1a48727b62305b778d8f6fbc3\": container with ID starting with f88b316f81ed9a1f1ac59afa00e48e5080b566e1a48727b62305b778d8f6fbc3 not found: ID does not exist" containerID="f88b316f81ed9a1f1ac59afa00e48e5080b566e1a48727b62305b778d8f6fbc3" Oct 31 00:09:02.212929 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:02.212858 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f88b316f81ed9a1f1ac59afa00e48e5080b566e1a48727b62305b778d8f6fbc3"} err="failed to get container status \"f88b316f81ed9a1f1ac59afa00e48e5080b566e1a48727b62305b778d8f6fbc3\": rpc error: code = NotFound desc = could not find container \"f88b316f81ed9a1f1ac59afa00e48e5080b566e1a48727b62305b778d8f6fbc3\": container with ID starting with f88b316f81ed9a1f1ac59afa00e48e5080b566e1a48727b62305b778d8f6fbc3 not found: ID does not exist" Oct 31 00:09:02.212929 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:02.212880 5274 scope.go:122] "RemoveContainer" containerID="c52b2ef878da6b12060990377a6804c5ba4388f48c2403866b3c0dc4dee7a551" Oct 31 00:09:02.213129 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:09:02.213113 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c52b2ef878da6b12060990377a6804c5ba4388f48c2403866b3c0dc4dee7a551\": container with ID starting with c52b2ef878da6b12060990377a6804c5ba4388f48c2403866b3c0dc4dee7a551 not found: ID does not exist" containerID="c52b2ef878da6b12060990377a6804c5ba4388f48c2403866b3c0dc4dee7a551" Oct 31 00:09:02.213168 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:02.213135 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c52b2ef878da6b12060990377a6804c5ba4388f48c2403866b3c0dc4dee7a551"} err="failed to get container status \"c52b2ef878da6b12060990377a6804c5ba4388f48c2403866b3c0dc4dee7a551\": rpc error: code = NotFound desc = could not find container \"c52b2ef878da6b12060990377a6804c5ba4388f48c2403866b3c0dc4dee7a551\": container with ID starting with c52b2ef878da6b12060990377a6804c5ba4388f48c2403866b3c0dc4dee7a551 not found: ID does not exist" Oct 31 00:09:02.213168 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:02.213149 5274 scope.go:122] "RemoveContainer" containerID="11b8de1f7bc9ccce7c8998af77fe871bc23ded88d6a4b80bd4b08333c79f8bb2" Oct 31 00:09:02.213386 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:09:02.213365 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11b8de1f7bc9ccce7c8998af77fe871bc23ded88d6a4b80bd4b08333c79f8bb2\": container with ID starting with 11b8de1f7bc9ccce7c8998af77fe871bc23ded88d6a4b80bd4b08333c79f8bb2 not found: ID does not exist" containerID="11b8de1f7bc9ccce7c8998af77fe871bc23ded88d6a4b80bd4b08333c79f8bb2" Oct 31 00:09:02.213443 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:02.213394 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11b8de1f7bc9ccce7c8998af77fe871bc23ded88d6a4b80bd4b08333c79f8bb2"} err="failed to get container status \"11b8de1f7bc9ccce7c8998af77fe871bc23ded88d6a4b80bd4b08333c79f8bb2\": rpc error: code = NotFound desc = could not find container \"11b8de1f7bc9ccce7c8998af77fe871bc23ded88d6a4b80bd4b08333c79f8bb2\": container with ID starting with 11b8de1f7bc9ccce7c8998af77fe871bc23ded88d6a4b80bd4b08333c79f8bb2 not found: ID does not exist" Oct 31 00:09:02.216858 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:02.216838 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:09:02.222536 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:02.222516 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:09:03.481177 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:03.481146 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4401656b-3d34-4bf5-956b-3a0fc555064f" path="/var/lib/kubelet/pods/4401656b-3d34-4bf5-956b-3a0fc555064f/volumes" Oct 31 00:09:08.558067 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.558026 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m2s4f"] Oct 31 00:09:08.558591 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.558548 5274 cpu_manager.go:414] "containerMap: removing container" podUID="4401656b-3d34-4bf5-956b-3a0fc555064f" containerName="tempo-gateway" Oct 31 00:09:08.558591 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.558567 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4401656b-3d34-4bf5-956b-3a0fc555064f" containerName="tempo-gateway" Oct 31 00:09:08.558709 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.558600 5274 cpu_manager.go:414] "containerMap: removing container" podUID="4401656b-3d34-4bf5-956b-3a0fc555064f" containerName="tempo-gateway-opa" Oct 31 00:09:08.558709 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.558608 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4401656b-3d34-4bf5-956b-3a0fc555064f" containerName="tempo-gateway-opa" Oct 31 00:09:08.558709 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.558622 5274 cpu_manager.go:414] "containerMap: removing container" podUID="3c64e116-a1aa-4fdd-bd37-b7ee8771e404" containerName="korrel8r" Oct 31 00:09:08.558709 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.558631 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c64e116-a1aa-4fdd-bd37-b7ee8771e404" containerName="korrel8r" Oct 31 00:09:08.558709 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.558641 5274 cpu_manager.go:414] "containerMap: removing container" podUID="4401656b-3d34-4bf5-956b-3a0fc555064f" containerName="tempo" Oct 31 00:09:08.558709 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.558649 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4401656b-3d34-4bf5-956b-3a0fc555064f" containerName="tempo" Oct 31 00:09:08.559029 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.558740 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="4401656b-3d34-4bf5-956b-3a0fc555064f" containerName="tempo-gateway" Oct 31 00:09:08.559029 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.558758 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="4401656b-3d34-4bf5-956b-3a0fc555064f" containerName="tempo-gateway-opa" Oct 31 00:09:08.559029 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.558769 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c64e116-a1aa-4fdd-bd37-b7ee8771e404" containerName="korrel8r" Oct 31 00:09:08.559029 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.558777 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="4401656b-3d34-4bf5-956b-3a0fc555064f" containerName="tempo" Oct 31 00:09:08.561853 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.561834 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m2s4f" Oct 31 00:09:08.569345 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.569297 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:09:08.571565 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.571541 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:09:08.572966 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.572935 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-2qhfb\"" Oct 31 00:09:08.573060 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.572945 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\"" Oct 31 00:09:08.573060 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.572941 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\"" Oct 31 00:09:08.574018 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.573998 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 31 00:09:08.576647 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.576304 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m2s4f"] Oct 31 00:09:08.615446 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.615415 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-2plj2"] Oct 31 00:09:08.619506 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.619487 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2plj2" Oct 31 00:09:08.627290 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.627264 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-2plj2"] Oct 31 00:09:08.668908 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.668880 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:09:08.668908 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.668905 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 31 00:09:08.672374 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.672359 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 31 00:09:08.672467 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.672394 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-s4j7m\"" Oct 31 00:09:08.720368 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.720336 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcbz9\" (UniqueName: \"kubernetes.io/projected/406bd387-d4a8-4388-a6dd-cc38c86d36fd-kube-api-access-tcbz9\") pod \"korrel8r-66d6f7f696-2plj2\" (UID: \"406bd387-d4a8-4388-a6dd-cc38c86d36fd\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2plj2" Oct 31 00:09:08.720494 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.720376 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f0a400e-5e95-4403-af53-6108a54263ca-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-m2s4f\" (UID: \"8f0a400e-5e95-4403-af53-6108a54263ca\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m2s4f" Oct 31 00:09:08.720550 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.720501 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/8f0a400e-5e95-4403-af53-6108a54263ca-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-m2s4f\" (UID: \"8f0a400e-5e95-4403-af53-6108a54263ca\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m2s4f" Oct 31 00:09:08.720550 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.720531 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/406bd387-d4a8-4388-a6dd-cc38c86d36fd-config\") pod \"korrel8r-66d6f7f696-2plj2\" (UID: \"406bd387-d4a8-4388-a6dd-cc38c86d36fd\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2plj2" Oct 31 00:09:08.720651 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.720592 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdw77\" (UniqueName: \"kubernetes.io/projected/8f0a400e-5e95-4403-af53-6108a54263ca-kube-api-access-xdw77\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-m2s4f\" (UID: \"8f0a400e-5e95-4403-af53-6108a54263ca\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m2s4f" Oct 31 00:09:08.720710 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.720696 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/406bd387-d4a8-4388-a6dd-cc38c86d36fd-tls\") pod \"korrel8r-66d6f7f696-2plj2\" (UID: \"406bd387-d4a8-4388-a6dd-cc38c86d36fd\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2plj2" Oct 31 00:09:08.720764 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.720723 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/8f0a400e-5e95-4403-af53-6108a54263ca-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-m2s4f\" (UID: \"8f0a400e-5e95-4403-af53-6108a54263ca\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m2s4f" Oct 31 00:09:08.720817 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.720759 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/8f0a400e-5e95-4403-af53-6108a54263ca-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-m2s4f\" (UID: \"8f0a400e-5e95-4403-af53-6108a54263ca\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m2s4f" Oct 31 00:09:08.822114 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.822036 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/406bd387-d4a8-4388-a6dd-cc38c86d36fd-tls\") pod \"korrel8r-66d6f7f696-2plj2\" (UID: \"406bd387-d4a8-4388-a6dd-cc38c86d36fd\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2plj2" Oct 31 00:09:08.822114 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.822086 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/8f0a400e-5e95-4403-af53-6108a54263ca-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-m2s4f\" (UID: \"8f0a400e-5e95-4403-af53-6108a54263ca\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m2s4f" Oct 31 00:09:08.822114 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.822107 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/8f0a400e-5e95-4403-af53-6108a54263ca-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-m2s4f\" (UID: \"8f0a400e-5e95-4403-af53-6108a54263ca\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m2s4f" Oct 31 00:09:08.822389 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.822138 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcbz9\" (UniqueName: \"kubernetes.io/projected/406bd387-d4a8-4388-a6dd-cc38c86d36fd-kube-api-access-tcbz9\") pod \"korrel8r-66d6f7f696-2plj2\" (UID: \"406bd387-d4a8-4388-a6dd-cc38c86d36fd\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2plj2" Oct 31 00:09:08.822389 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.822155 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f0a400e-5e95-4403-af53-6108a54263ca-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-m2s4f\" (UID: \"8f0a400e-5e95-4403-af53-6108a54263ca\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m2s4f" Oct 31 00:09:08.822389 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.822186 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/8f0a400e-5e95-4403-af53-6108a54263ca-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-m2s4f\" (UID: \"8f0a400e-5e95-4403-af53-6108a54263ca\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m2s4f" Oct 31 00:09:08.822389 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.822205 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/406bd387-d4a8-4388-a6dd-cc38c86d36fd-config\") pod \"korrel8r-66d6f7f696-2plj2\" (UID: \"406bd387-d4a8-4388-a6dd-cc38c86d36fd\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2plj2" Oct 31 00:09:08.822389 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.822223 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdw77\" (UniqueName: \"kubernetes.io/projected/8f0a400e-5e95-4403-af53-6108a54263ca-kube-api-access-xdw77\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-m2s4f\" (UID: \"8f0a400e-5e95-4403-af53-6108a54263ca\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m2s4f" Oct 31 00:09:08.822389 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:09:08.822228 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/korrel8r-tls: secret "korrel8r-tls" not found Oct 31 00:09:08.822389 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:09:08.822310 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/406bd387-d4a8-4388-a6dd-cc38c86d36fd-tls podName:406bd387-d4a8-4388-a6dd-cc38c86d36fd nodeName:}" failed. No retries permitted until 2026-10-31 00:09:09.322288063 +0000 UTC m=+585.959106776 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls" (UniqueName: "kubernetes.io/secret/406bd387-d4a8-4388-a6dd-cc38c86d36fd-tls") pod "korrel8r-66d6f7f696-2plj2" (UID: "406bd387-d4a8-4388-a6dd-cc38c86d36fd") : secret "korrel8r-tls" not found Oct 31 00:09:08.822389 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:09:08.822317 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found Oct 31 00:09:08.822389 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:09:08.822391 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f0a400e-5e95-4403-af53-6108a54263ca-prometheus-client-cert podName:8f0a400e-5e95-4403-af53-6108a54263ca nodeName:}" failed. No retries permitted until 2026-10-31 00:09:09.322373922 +0000 UTC m=+585.959192639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/8f0a400e-5e95-4403-af53-6108a54263ca-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-m2s4f" (UID: "8f0a400e-5e95-4403-af53-6108a54263ca") : secret "prometheus-operated-tls" not found Oct 31 00:09:08.822898 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.822878 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/8f0a400e-5e95-4403-af53-6108a54263ca-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-m2s4f\" (UID: \"8f0a400e-5e95-4403-af53-6108a54263ca\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m2s4f" Oct 31 00:09:08.823108 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.823088 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/406bd387-d4a8-4388-a6dd-cc38c86d36fd-config\") pod \"korrel8r-66d6f7f696-2plj2\" (UID: \"406bd387-d4a8-4388-a6dd-cc38c86d36fd\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2plj2" Oct 31 00:09:08.824692 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.824667 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/8f0a400e-5e95-4403-af53-6108a54263ca-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-m2s4f\" (UID: \"8f0a400e-5e95-4403-af53-6108a54263ca\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m2s4f" Oct 31 00:09:08.824784 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.824711 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f0a400e-5e95-4403-af53-6108a54263ca-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-m2s4f\" (UID: \"8f0a400e-5e95-4403-af53-6108a54263ca\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m2s4f" Oct 31 00:09:08.836638 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.836613 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcbz9\" (UniqueName: \"kubernetes.io/projected/406bd387-d4a8-4388-a6dd-cc38c86d36fd-kube-api-access-tcbz9\") pod \"korrel8r-66d6f7f696-2plj2\" (UID: \"406bd387-d4a8-4388-a6dd-cc38c86d36fd\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2plj2" Oct 31 00:09:08.837759 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:08.837738 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdw77\" (UniqueName: \"kubernetes.io/projected/8f0a400e-5e95-4403-af53-6108a54263ca-kube-api-access-xdw77\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-m2s4f\" (UID: \"8f0a400e-5e95-4403-af53-6108a54263ca\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m2s4f" Oct 31 00:09:09.329034 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:09.329000 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/406bd387-d4a8-4388-a6dd-cc38c86d36fd-tls\") pod \"korrel8r-66d6f7f696-2plj2\" (UID: \"406bd387-d4a8-4388-a6dd-cc38c86d36fd\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2plj2" Oct 31 00:09:09.329034 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:09.329039 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/8f0a400e-5e95-4403-af53-6108a54263ca-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-m2s4f\" (UID: \"8f0a400e-5e95-4403-af53-6108a54263ca\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m2s4f" Oct 31 00:09:09.331430 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:09.331395 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/406bd387-d4a8-4388-a6dd-cc38c86d36fd-tls\") pod \"korrel8r-66d6f7f696-2plj2\" (UID: \"406bd387-d4a8-4388-a6dd-cc38c86d36fd\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2plj2" Oct 31 00:09:09.331572 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:09.331553 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/8f0a400e-5e95-4403-af53-6108a54263ca-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-m2s4f\" (UID: \"8f0a400e-5e95-4403-af53-6108a54263ca\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m2s4f" Oct 31 00:09:09.474693 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:09.474665 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m2s4f" Oct 31 00:09:09.533483 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:09.533295 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2plj2" Oct 31 00:09:09.754842 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:09.754818 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m2s4f"] Oct 31 00:09:09.757253 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:09:09.757229 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f0a400e_5e95_4403_af53_6108a54263ca.slice/crio-53635424bc47a858c1058335782ad6faae6cfdc57eed40704ab69b0f84a5e171 WatchSource:0}: Error finding container 53635424bc47a858c1058335782ad6faae6cfdc57eed40704ab69b0f84a5e171: Status 404 returned error can't find the container with id 53635424bc47a858c1058335782ad6faae6cfdc57eed40704ab69b0f84a5e171 Oct 31 00:09:09.821593 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:09.821572 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-2plj2"] Oct 31 00:09:09.823573 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:09:09.823544 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod406bd387_d4a8_4388_a6dd_cc38c86d36fd.slice/crio-bdf366ad3a66d877c041f14ebe46cb1458a8cf7a01cf20314cbee795f88e3b2a WatchSource:0}: Error finding container bdf366ad3a66d877c041f14ebe46cb1458a8cf7a01cf20314cbee795f88e3b2a: Status 404 returned error can't find the container with id bdf366ad3a66d877c041f14ebe46cb1458a8cf7a01cf20314cbee795f88e3b2a Oct 31 00:09:10.132729 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.132697 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:09:10.136894 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.136872 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:10.141932 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.141903 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\"" Oct 31 00:09:10.141932 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.141923 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 31 00:09:10.142093 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.141949 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\"" Oct 31 00:09:10.143235 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.143213 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\"" Oct 31 00:09:10.143379 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.143238 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-f9hjt\"" Oct 31 00:09:10.143471 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.143221 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\"" Oct 31 00:09:10.143471 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.143254 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\"" Oct 31 00:09:10.143471 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.143322 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 31 00:09:10.154505 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.154480 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:09:10.230873 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.230840 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2plj2" event={"ID":"406bd387-d4a8-4388-a6dd-cc38c86d36fd","Type":"ContainerStarted","Data":"3cae132b592b809f49d189e7c23d221d1d3019c4beaccad5817fd8db1d59cb16"} Oct 31 00:09:10.231061 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.230900 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2plj2" event={"ID":"406bd387-d4a8-4388-a6dd-cc38c86d36fd","Type":"ContainerStarted","Data":"bdf366ad3a66d877c041f14ebe46cb1458a8cf7a01cf20314cbee795f88e3b2a"} Oct 31 00:09:10.231061 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.230922 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2plj2" Oct 31 00:09:10.232066 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.232044 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m2s4f" event={"ID":"8f0a400e-5e95-4403-af53-6108a54263ca","Type":"ContainerStarted","Data":"53635424bc47a858c1058335782ad6faae6cfdc57eed40704ab69b0f84a5e171"} Oct 31 00:09:10.238004 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.237981 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/6885e4a3-3f18-477a-90ef-f59dbef4d404-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:10.238090 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.238015 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/6885e4a3-3f18-477a-90ef-f59dbef4d404-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:10.238090 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.238035 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/6885e4a3-3f18-477a-90ef-f59dbef4d404-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:10.238090 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.238051 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6885e4a3-3f18-477a-90ef-f59dbef4d404-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:10.238231 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.238160 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/6885e4a3-3f18-477a-90ef-f59dbef4d404-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:10.238231 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.238199 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfm2t\" (UniqueName: \"kubernetes.io/projected/6885e4a3-3f18-477a-90ef-f59dbef4d404-kube-api-access-qfm2t\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:10.238231 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.238221 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/6885e4a3-3f18-477a-90ef-f59dbef4d404-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:10.238322 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.238243 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/6885e4a3-3f18-477a-90ef-f59dbef4d404-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:10.250569 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.250520 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2plj2" podStartSLOduration=2.250507581 podStartE2EDuration="2.250507581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:10.249484387 +0000 UTC m=+586.886303104" watchObservedRunningTime="2026-10-31 00:09:10.250507581 +0000 UTC m=+586.887326297" Oct 31 00:09:10.338900 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.338876 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfm2t\" (UniqueName: \"kubernetes.io/projected/6885e4a3-3f18-477a-90ef-f59dbef4d404-kube-api-access-qfm2t\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:10.339051 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.338907 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/6885e4a3-3f18-477a-90ef-f59dbef4d404-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:10.339051 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.338930 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/6885e4a3-3f18-477a-90ef-f59dbef4d404-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:10.339167 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.339123 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/6885e4a3-3f18-477a-90ef-f59dbef4d404-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:10.339236 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.339174 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/6885e4a3-3f18-477a-90ef-f59dbef4d404-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:10.339236 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.339206 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/6885e4a3-3f18-477a-90ef-f59dbef4d404-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:10.339346 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.339240 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6885e4a3-3f18-477a-90ef-f59dbef4d404-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:10.339346 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.339297 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/6885e4a3-3f18-477a-90ef-f59dbef4d404-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:10.339722 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.339700 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/6885e4a3-3f18-477a-90ef-f59dbef4d404-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:10.339888 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.339861 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/6885e4a3-3f18-477a-90ef-f59dbef4d404-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:10.340162 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.340138 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6885e4a3-3f18-477a-90ef-f59dbef4d404-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:10.340233 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.340192 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/6885e4a3-3f18-477a-90ef-f59dbef4d404-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:10.342128 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.341937 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/6885e4a3-3f18-477a-90ef-f59dbef4d404-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:10.342241 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.342181 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/6885e4a3-3f18-477a-90ef-f59dbef4d404-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:10.342326 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.342304 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/6885e4a3-3f18-477a-90ef-f59dbef4d404-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:10.351340 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:10.351318 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfm2t\" (UniqueName: \"kubernetes.io/projected/6885e4a3-3f18-477a-90ef-f59dbef4d404-kube-api-access-qfm2t\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:11.243763 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:11.243722 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m2s4f" event={"ID":"8f0a400e-5e95-4403-af53-6108a54263ca","Type":"ContainerStarted","Data":"30e2112ae4ba567c5e78dc7b967fea51e8c179ea7a7d52b7089711469e86f8ae"} Oct 31 00:09:11.244175 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:11.244074 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m2s4f" Oct 31 00:09:11.265214 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:11.265161 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m2s4f" podStartSLOduration=3.265145913 podStartE2EDuration="3.265145913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:11.26298707 +0000 UTC m=+587.899805789" watchObservedRunningTime="2026-10-31 00:09:11.265145913 +0000 UTC m=+587.901964629" Oct 31 00:09:15.047938 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.047888 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:09:15.051624 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.051609 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:09:15.057901 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.057875 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 31 00:09:15.058039 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.057930 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 31 00:09:15.058039 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.057991 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:09:15.058372 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.058357 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 31 00:09:15.058480 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.058458 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-wvv2m\"" Oct 31 00:09:15.064837 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.064816 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:09:15.183162 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.183131 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c14cc7ba-1add-4829-806a-639de0bc93eb-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c14cc7ba-1add-4829-806a-639de0bc93eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:09:15.183313 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.183166 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/c14cc7ba-1add-4829-806a-639de0bc93eb-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c14cc7ba-1add-4829-806a-639de0bc93eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:09:15.183313 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.183194 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljddk\" (UniqueName: \"kubernetes.io/projected/c14cc7ba-1add-4829-806a-639de0bc93eb-kube-api-access-ljddk\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c14cc7ba-1add-4829-806a-639de0bc93eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:09:15.183382 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.183328 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c14cc7ba-1add-4829-806a-639de0bc93eb-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c14cc7ba-1add-4829-806a-639de0bc93eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:09:15.183382 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.183361 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c14cc7ba-1add-4829-806a-639de0bc93eb-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c14cc7ba-1add-4829-806a-639de0bc93eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:09:15.183445 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.183391 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c14cc7ba-1add-4829-806a-639de0bc93eb-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c14cc7ba-1add-4829-806a-639de0bc93eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:09:15.183477 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.183466 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c14cc7ba-1add-4829-806a-639de0bc93eb-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c14cc7ba-1add-4829-806a-639de0bc93eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:09:15.250074 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.250046 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:09:15.254046 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.254024 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:15.258115 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.258067 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 31 00:09:15.258115 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.258108 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:09:15.258339 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.258318 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 31 00:09:15.258339 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.258330 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-46rq9\"" Oct 31 00:09:15.258461 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.258398 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 31 00:09:15.258511 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.258479 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:09:15.271930 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.271907 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:09:15.284562 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.284538 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c14cc7ba-1add-4829-806a-639de0bc93eb-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c14cc7ba-1add-4829-806a-639de0bc93eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:09:15.284645 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.284569 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/c14cc7ba-1add-4829-806a-639de0bc93eb-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c14cc7ba-1add-4829-806a-639de0bc93eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:09:15.284645 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.284591 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljddk\" (UniqueName: \"kubernetes.io/projected/c14cc7ba-1add-4829-806a-639de0bc93eb-kube-api-access-ljddk\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c14cc7ba-1add-4829-806a-639de0bc93eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:09:15.284737 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.284641 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c14cc7ba-1add-4829-806a-639de0bc93eb-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c14cc7ba-1add-4829-806a-639de0bc93eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:09:15.284737 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.284692 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c14cc7ba-1add-4829-806a-639de0bc93eb-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c14cc7ba-1add-4829-806a-639de0bc93eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:09:15.284833 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.284738 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c14cc7ba-1add-4829-806a-639de0bc93eb-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c14cc7ba-1add-4829-806a-639de0bc93eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:09:15.284833 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.284793 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c14cc7ba-1add-4829-806a-639de0bc93eb-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c14cc7ba-1add-4829-806a-639de0bc93eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:09:15.285546 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.285487 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/c14cc7ba-1add-4829-806a-639de0bc93eb-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c14cc7ba-1add-4829-806a-639de0bc93eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:09:15.287684 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.287662 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c14cc7ba-1add-4829-806a-639de0bc93eb-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c14cc7ba-1add-4829-806a-639de0bc93eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:09:15.287859 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.287841 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c14cc7ba-1add-4829-806a-639de0bc93eb-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c14cc7ba-1add-4829-806a-639de0bc93eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:09:15.287906 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.287878 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c14cc7ba-1add-4829-806a-639de0bc93eb-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c14cc7ba-1add-4829-806a-639de0bc93eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:09:15.287976 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.287932 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c14cc7ba-1add-4829-806a-639de0bc93eb-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c14cc7ba-1add-4829-806a-639de0bc93eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:09:15.288256 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.288240 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c14cc7ba-1add-4829-806a-639de0bc93eb-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c14cc7ba-1add-4829-806a-639de0bc93eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:09:15.296706 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.296677 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljddk\" (UniqueName: \"kubernetes.io/projected/c14cc7ba-1add-4829-806a-639de0bc93eb-kube-api-access-ljddk\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c14cc7ba-1add-4829-806a-639de0bc93eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:09:15.337495 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.337405 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 31 00:09:15.337495 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.337420 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 31 00:09:15.337626 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.337529 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 31 00:09:15.369898 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.369875 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:09:15.385295 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.385260 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6dd882e7-9623-45d4-96f1-0cd002fb7906-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:15.385405 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.385306 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:15.385472 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.385420 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6dd882e7-9623-45d4-96f1-0cd002fb7906-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:15.385537 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.385470 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6dd882e7-9623-45d4-96f1-0cd002fb7906-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:15.385537 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.385497 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:15.385646 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.385617 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6dd882e7-9623-45d4-96f1-0cd002fb7906-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:15.385704 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.385654 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6dd882e7-9623-45d4-96f1-0cd002fb7906-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:15.385772 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.385748 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:15.385842 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.385792 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnthd\" (UniqueName: \"kubernetes.io/projected/6dd882e7-9623-45d4-96f1-0cd002fb7906-kube-api-access-mnthd\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:15.385907 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.385850 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:15.385983 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.385922 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:15.386366 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.386030 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:15.487375 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.487345 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:15.487548 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.487394 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6dd882e7-9623-45d4-96f1-0cd002fb7906-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:15.487548 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.487412 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:15.487548 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.487438 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6dd882e7-9623-45d4-96f1-0cd002fb7906-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:15.487548 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.487472 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6dd882e7-9623-45d4-96f1-0cd002fb7906-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:15.487548 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.487490 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:15.487548 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.487507 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6dd882e7-9623-45d4-96f1-0cd002fb7906-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:15.487548 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.487536 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6dd882e7-9623-45d4-96f1-0cd002fb7906-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:15.487892 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.487564 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:15.487892 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.487610 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnthd\" (UniqueName: \"kubernetes.io/projected/6dd882e7-9623-45d4-96f1-0cd002fb7906-kube-api-access-mnthd\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:15.487892 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.487671 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:15.487892 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.487703 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:15.488158 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.488111 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6dd882e7-9623-45d4-96f1-0cd002fb7906-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:15.488518 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.488486 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6dd882e7-9623-45d4-96f1-0cd002fb7906-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:15.488703 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.488651 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6dd882e7-9623-45d4-96f1-0cd002fb7906-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:15.490689 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.490655 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6dd882e7-9623-45d4-96f1-0cd002fb7906-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:15.491240 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.490744 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:15.491240 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.490865 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:15.491403 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.491177 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:15.491403 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.491393 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:15.491600 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.491569 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:15.491687 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.491606 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:15.491722 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.491696 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6dd882e7-9623-45d4-96f1-0cd002fb7906-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:15.500427 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.500408 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnthd\" (UniqueName: \"kubernetes.io/projected/6dd882e7-9623-45d4-96f1-0cd002fb7906-kube-api-access-mnthd\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:15.656029 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:15.656004 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:09:15.657655 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:09:15.657628 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc14cc7ba_1add_4829_806a_639de0bc93eb.slice/crio-3b78481c14a5ff6f1d214e0152c2de901e34c033c2efda66165873ddcfb7e509 WatchSource:0}: Error finding container 3b78481c14a5ff6f1d214e0152c2de901e34c033c2efda66165873ddcfb7e509: Status 404 returned error can't find the container with id 3b78481c14a5ff6f1d214e0152c2de901e34c033c2efda66165873ddcfb7e509 Oct 31 00:09:16.193523 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:16.193490 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:16.200402 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:16.200372 5274 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:16.269890 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:16.269857 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"c14cc7ba-1add-4829-806a-639de0bc93eb","Type":"ContainerStarted","Data":"b65cd7e92f819ee6860d78f0d097f3d94a34f86f9f694281d5dd8c91fc0e812d"} Oct 31 00:09:16.270095 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:16.269922 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"c14cc7ba-1add-4829-806a-639de0bc93eb","Type":"ContainerStarted","Data":"3b78481c14a5ff6f1d214e0152c2de901e34c033c2efda66165873ddcfb7e509"} Oct 31 00:09:16.294210 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:16.294176 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:16.294391 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:16.294321 5274 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:16.300118 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:16.300098 5274 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") DevicePath \"csi-9ce3c80b5d5deac9dfdb953db4fb017080b654a85db76a2937e67cd541d89057\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:16.389341 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:16.389312 5274 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/151619720cb39d40781bf258206fb0be746b77e5f55879e189e4cf0d6e46dfb0/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:16.404891 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:16.404868 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:16.455575 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:16.455552 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:16.735262 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:16.735168 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:09:16.737382 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:09:16.737353 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6885e4a3_3f18_477a_90ef_f59dbef4d404.slice/crio-552c524b296d04c810417b3b056cb2eac35303c63b2b364b05eb6660b177a7c9 WatchSource:0}: Error finding container 552c524b296d04c810417b3b056cb2eac35303c63b2b364b05eb6660b177a7c9: Status 404 returned error can't find the container with id 552c524b296d04c810417b3b056cb2eac35303c63b2b364b05eb6660b177a7c9 Oct 31 00:09:17.250812 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:17.250780 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m2s4f" Oct 31 00:09:17.279392 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:17.279350 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"6885e4a3-3f18-477a-90ef-f59dbef4d404","Type":"ContainerStarted","Data":"db4bad30802608be7167243b935f595c692c98e9a2700be82e79a0e2173581af"} Oct 31 00:09:17.279584 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:17.279411 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"6885e4a3-3f18-477a-90ef-f59dbef4d404","Type":"ContainerStarted","Data":"0999e1bfd89f852a24dd8bc572c374cce7bb0b18b8f7d475cd96a3cc548e1229"} Oct 31 00:09:17.279584 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:17.279432 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"6885e4a3-3f18-477a-90ef-f59dbef4d404","Type":"ContainerStarted","Data":"eff1b373ae1994af65d6c6cdde9b0a7c7fd4a2524c98d0c10dd6e9f198178298"} Oct 31 00:09:17.279584 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:17.279449 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"6885e4a3-3f18-477a-90ef-f59dbef4d404","Type":"ContainerStarted","Data":"552c524b296d04c810417b3b056cb2eac35303c63b2b364b05eb6660b177a7c9"} Oct 31 00:09:17.305191 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:17.305146 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=7.305132024 podStartE2EDuration="7.305132024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:09:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:17.304140976 +0000 UTC m=+593.940959693" watchObservedRunningTime="2026-10-31 00:09:17.305132024 +0000 UTC m=+593.941950739" Oct 31 00:09:17.707695 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:17.707664 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:17.715181 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:17.715153 5274 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:17.808531 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:17.808492 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:17.808690 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:17.808612 5274 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:17.812043 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:17.812025 5274 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") DevicePath \"csi-3beb91b3ebf177571ca474589da3560cd18787cfa50b2a0b9bfef85661e6cf21\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:17.899722 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:17.899685 5274 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/534eb232558c8505fa55558aa87cc96e2d2b0adb946edb717939ec387be54e97/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:17.915258 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:17.915234 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:17.974803 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:17.974752 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:18.258825 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:18.258797 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:09:18.261138 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:09:18.261106 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dd882e7_9623_45d4_96f1_0cd002fb7906.slice/crio-147dc4175ab1531e540032011e51ccef7f164da31f2b49fcaefd1843776ac519 WatchSource:0}: Error finding container 147dc4175ab1531e540032011e51ccef7f164da31f2b49fcaefd1843776ac519: Status 404 returned error can't find the container with id 147dc4175ab1531e540032011e51ccef7f164da31f2b49fcaefd1843776ac519 Oct 31 00:09:18.291615 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:18.291565 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"6dd882e7-9623-45d4-96f1-0cd002fb7906","Type":"ContainerStarted","Data":"147dc4175ab1531e540032011e51ccef7f164da31f2b49fcaefd1843776ac519"} Oct 31 00:09:18.291845 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:18.291824 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:18.292009 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:18.291982 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:18.292135 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:18.292036 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:18.294681 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:18.294660 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:18.300949 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:18.300930 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:09:19.298076 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:19.298039 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"6dd882e7-9623-45d4-96f1-0cd002fb7906","Type":"ContainerStarted","Data":"9ee773905955211ce11f090799d5253828696f1b1b383b5adffbb8e23f3faca0"} Oct 31 00:09:20.304122 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:20.304089 5274 generic.go:356] "Generic (PLEG): container finished" podID="c14cc7ba-1add-4829-806a-639de0bc93eb" containerID="b65cd7e92f819ee6860d78f0d097f3d94a34f86f9f694281d5dd8c91fc0e812d" exitCode=0 Oct 31 00:09:20.304527 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:20.304159 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"c14cc7ba-1add-4829-806a-639de0bc93eb","Type":"ContainerDied","Data":"b65cd7e92f819ee6860d78f0d097f3d94a34f86f9f694281d5dd8c91fc0e812d"} Oct 31 00:09:20.898650 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:09:20.898617 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:09:21.246133 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:21.246104 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2plj2" Oct 31 00:09:21.311640 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:21.311610 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"c14cc7ba-1add-4829-806a-639de0bc93eb","Type":"ContainerStarted","Data":"4dc0fed8214bf715d83257e4c8107ab14e73f84c779bb28d9a64c2319670e0b7"} Oct 31 00:09:21.312026 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:21.311667 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"c14cc7ba-1add-4829-806a-639de0bc93eb","Type":"ContainerStarted","Data":"3adcf308fd53cf1fa56b17d3c6935c8c0ddc1b5b21b5f6e6c0ed6bf6ee17ae3c"} Oct 31 00:09:21.312026 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:21.311835 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:09:21.340161 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:21.340098 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=2.9545990460000002 podStartE2EDuration="6.340084105s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.385485059s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:21.337842604 +0000 UTC m=+597.974661321" watchObservedRunningTime="2026-10-31 00:09:21.340084105 +0000 UTC m=+597.976902821" Oct 31 00:09:22.318474 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:22.318442 5274 generic.go:356] "Generic (PLEG): container finished" podID="6dd882e7-9623-45d4-96f1-0cd002fb7906" containerID="9ee773905955211ce11f090799d5253828696f1b1b383b5adffbb8e23f3faca0" exitCode=0 Oct 31 00:09:22.318932 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:22.318518 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"6dd882e7-9623-45d4-96f1-0cd002fb7906","Type":"ContainerDied","Data":"9ee773905955211ce11f090799d5253828696f1b1b383b5adffbb8e23f3faca0"} Oct 31 00:09:23.325965 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:23.325927 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"6dd882e7-9623-45d4-96f1-0cd002fb7906","Type":"ContainerStarted","Data":"999e2aaccd33e5183a4c655b8ce6ab5eceab442612f98fae15c8fa7e8e8268a9"} Oct 31 00:09:23.326330 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:23.326045 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"6dd882e7-9623-45d4-96f1-0cd002fb7906","Type":"ContainerStarted","Data":"6bdde96d361cbd4ea3cb2cca44731a726743511d45dbf6ad661524b43ffcdb14"} Oct 31 00:09:23.326330 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:23.326077 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"6dd882e7-9623-45d4-96f1-0cd002fb7906","Type":"ContainerStarted","Data":"5ffa85c5e5daa8106bc8ea56a7939efa6abdda48e89272ea6b4ced9f898fcc18"} Oct 31 00:09:23.362082 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:23.362029 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=4.777074758 podStartE2EDuration="8.362013594s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.584938836s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:23.360100876 +0000 UTC m=+599.996919592" watchObservedRunningTime="2026-10-31 00:09:23.362013594 +0000 UTC m=+599.998832310" Oct 31 00:09:27.319593 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:27.319557 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:09:27.975123 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:27.975086 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:32.975525 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:32.975492 5274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:32.980591 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:32.980572 5274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:33.381562 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:33.381536 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:09:40.305844 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:40.305797 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="6885e4a3-3f18-477a-90ef-f59dbef4d404" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:09:50.306077 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:09:50.306039 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="6885e4a3-3f18-477a-90ef-f59dbef4d404" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:09:50.900273 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:09:50.900239 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:10:00.305880 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:00.305849 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:10:01.359788 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:01.359749 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m2s4f"] Oct 31 00:10:01.360222 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:01.360048 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m2s4f" podUID="8f0a400e-5e95-4403-af53-6108a54263ca" containerName="kube-rbac-proxy" containerID="cri-o://30e2112ae4ba567c5e78dc7b967fea51e8c179ea7a7d52b7089711469e86f8ae" gracePeriod=30 Oct 31 00:10:01.375664 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:01.375641 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-2plj2"] Oct 31 00:10:01.375879 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:01.375812 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2plj2" podUID="406bd387-d4a8-4388-a6dd-cc38c86d36fd" containerName="korrel8r" containerID="cri-o://3cae132b592b809f49d189e7c23d221d1d3019c4beaccad5817fd8db1d59cb16" gracePeriod=30 Oct 31 00:10:01.525208 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:01.525167 5274 generic.go:356] "Generic (PLEG): container finished" podID="406bd387-d4a8-4388-a6dd-cc38c86d36fd" containerID="3cae132b592b809f49d189e7c23d221d1d3019c4beaccad5817fd8db1d59cb16" exitCode=0 Oct 31 00:10:01.525380 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:01.525309 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2plj2" event={"ID":"406bd387-d4a8-4388-a6dd-cc38c86d36fd","Type":"ContainerDied","Data":"3cae132b592b809f49d189e7c23d221d1d3019c4beaccad5817fd8db1d59cb16"} Oct 31 00:10:01.529664 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:01.528739 5274 generic.go:356] "Generic (PLEG): container finished" podID="8f0a400e-5e95-4403-af53-6108a54263ca" containerID="30e2112ae4ba567c5e78dc7b967fea51e8c179ea7a7d52b7089711469e86f8ae" exitCode=0 Oct 31 00:10:01.529664 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:01.528866 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m2s4f" event={"ID":"8f0a400e-5e95-4403-af53-6108a54263ca","Type":"ContainerDied","Data":"30e2112ae4ba567c5e78dc7b967fea51e8c179ea7a7d52b7089711469e86f8ae"} Oct 31 00:10:01.574682 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:01.574656 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:10:01.575469 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:01.575268 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="6dd882e7-9623-45d4-96f1-0cd002fb7906" containerName="thanos-sidecar" containerID="cri-o://999e2aaccd33e5183a4c655b8ce6ab5eceab442612f98fae15c8fa7e8e8268a9" gracePeriod=600 Oct 31 00:10:01.575469 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:01.575404 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="6dd882e7-9623-45d4-96f1-0cd002fb7906" containerName="config-reloader" containerID="cri-o://6bdde96d361cbd4ea3cb2cca44731a726743511d45dbf6ad661524b43ffcdb14" gracePeriod=600 Oct 31 00:10:01.576762 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:01.576741 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="6dd882e7-9623-45d4-96f1-0cd002fb7906" containerName="prometheus" containerID="cri-o://5ffa85c5e5daa8106bc8ea56a7939efa6abdda48e89272ea6b4ced9f898fcc18" gracePeriod=600 Oct 31 00:10:01.581825 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:01.580037 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:10:01.581825 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:01.580288 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="c14cc7ba-1add-4829-806a-639de0bc93eb" containerName="alertmanager" containerID="cri-o://3adcf308fd53cf1fa56b17d3c6935c8c0ddc1b5b21b5f6e6c0ed6bf6ee17ae3c" gracePeriod=120 Oct 31 00:10:01.581825 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:01.580412 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="c14cc7ba-1add-4829-806a-639de0bc93eb" containerName="config-reloader" containerID="cri-o://4dc0fed8214bf715d83257e4c8107ab14e73f84c779bb28d9a64c2319670e0b7" gracePeriod=120 Oct 31 00:10:01.734680 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:01.734197 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2plj2" Oct 31 00:10:01.737080 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:01.737055 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m2s4f" Oct 31 00:10:01.902135 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:01.901445 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f0a400e-5e95-4403-af53-6108a54263ca-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/8f0a400e-5e95-4403-af53-6108a54263ca-prometheus-client-cert\") pod \"8f0a400e-5e95-4403-af53-6108a54263ca\" (UID: \"8f0a400e-5e95-4403-af53-6108a54263ca\") " Oct 31 00:10:01.902290 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:01.902153 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f0a400e-5e95-4403-af53-6108a54263ca-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f0a400e-5e95-4403-af53-6108a54263ca-kube-rbac-proxy-config\") pod \"8f0a400e-5e95-4403-af53-6108a54263ca\" (UID: \"8f0a400e-5e95-4403-af53-6108a54263ca\") " Oct 31 00:10:01.902290 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:01.902196 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f0a400e-5e95-4403-af53-6108a54263ca-kube-api-access-xdw77\" (UniqueName: \"kubernetes.io/projected/8f0a400e-5e95-4403-af53-6108a54263ca-kube-api-access-xdw77\") pod \"8f0a400e-5e95-4403-af53-6108a54263ca\" (UID: \"8f0a400e-5e95-4403-af53-6108a54263ca\") " Oct 31 00:10:01.902290 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:01.902239 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f0a400e-5e95-4403-af53-6108a54263ca-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/8f0a400e-5e95-4403-af53-6108a54263ca-prometheus-ca\") pod \"8f0a400e-5e95-4403-af53-6108a54263ca\" (UID: \"8f0a400e-5e95-4403-af53-6108a54263ca\") " Oct 31 00:10:01.902290 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:01.902283 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/406bd387-d4a8-4388-a6dd-cc38c86d36fd-config\" (UniqueName: \"kubernetes.io/configmap/406bd387-d4a8-4388-a6dd-cc38c86d36fd-config\") pod \"406bd387-d4a8-4388-a6dd-cc38c86d36fd\" (UID: \"406bd387-d4a8-4388-a6dd-cc38c86d36fd\") " Oct 31 00:10:01.902519 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:01.902308 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f0a400e-5e95-4403-af53-6108a54263ca-tls\" (UniqueName: \"kubernetes.io/secret/8f0a400e-5e95-4403-af53-6108a54263ca-tls\") pod \"8f0a400e-5e95-4403-af53-6108a54263ca\" (UID: \"8f0a400e-5e95-4403-af53-6108a54263ca\") " Oct 31 00:10:01.902519 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:01.902360 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/406bd387-d4a8-4388-a6dd-cc38c86d36fd-kube-api-access-tcbz9\" (UniqueName: \"kubernetes.io/projected/406bd387-d4a8-4388-a6dd-cc38c86d36fd-kube-api-access-tcbz9\") pod \"406bd387-d4a8-4388-a6dd-cc38c86d36fd\" (UID: \"406bd387-d4a8-4388-a6dd-cc38c86d36fd\") " Oct 31 00:10:01.902519 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:01.902442 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/406bd387-d4a8-4388-a6dd-cc38c86d36fd-tls\" (UniqueName: \"kubernetes.io/secret/406bd387-d4a8-4388-a6dd-cc38c86d36fd-tls\") pod \"406bd387-d4a8-4388-a6dd-cc38c86d36fd\" (UID: \"406bd387-d4a8-4388-a6dd-cc38c86d36fd\") " Oct 31 00:10:01.902790 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:01.902767 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/406bd387-d4a8-4388-a6dd-cc38c86d36fd-config" pod "406bd387-d4a8-4388-a6dd-cc38c86d36fd" (UID: "406bd387-d4a8-4388-a6dd-cc38c86d36fd"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:10:01.904049 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:01.904008 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f0a400e-5e95-4403-af53-6108a54263ca-prometheus-ca" pod "8f0a400e-5e95-4403-af53-6108a54263ca" (UID: "8f0a400e-5e95-4403-af53-6108a54263ca"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:10:01.910533 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:01.910482 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f0a400e-5e95-4403-af53-6108a54263ca-kube-rbac-proxy-config" pod "8f0a400e-5e95-4403-af53-6108a54263ca" (UID: "8f0a400e-5e95-4403-af53-6108a54263ca"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:01.912308 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:01.911078 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f0a400e-5e95-4403-af53-6108a54263ca-prometheus-client-cert" pod "8f0a400e-5e95-4403-af53-6108a54263ca" (UID: "8f0a400e-5e95-4403-af53-6108a54263ca"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:01.912308 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:01.912063 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f0a400e-5e95-4403-af53-6108a54263ca-kube-api-access-xdw77" pod "8f0a400e-5e95-4403-af53-6108a54263ca" (UID: "8f0a400e-5e95-4403-af53-6108a54263ca"). InnerVolumeSpecName "kube-api-access-xdw77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:01.912607 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:01.912301 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f0a400e-5e95-4403-af53-6108a54263ca-tls" pod "8f0a400e-5e95-4403-af53-6108a54263ca" (UID: "8f0a400e-5e95-4403-af53-6108a54263ca"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:01.913934 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:01.913902 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/406bd387-d4a8-4388-a6dd-cc38c86d36fd-tls" pod "406bd387-d4a8-4388-a6dd-cc38c86d36fd" (UID: "406bd387-d4a8-4388-a6dd-cc38c86d36fd"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:01.914182 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:01.914149 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/406bd387-d4a8-4388-a6dd-cc38c86d36fd-kube-api-access-tcbz9" pod "406bd387-d4a8-4388-a6dd-cc38c86d36fd" (UID: "406bd387-d4a8-4388-a6dd-cc38c86d36fd"). InnerVolumeSpecName "kube-api-access-tcbz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:01.914368 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:01.914343 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:10:01.914818 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:01.914574 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="6885e4a3-3f18-477a-90ef-f59dbef4d404" containerName="tempo" containerID="cri-o://eff1b373ae1994af65d6c6cdde9b0a7c7fd4a2524c98d0c10dd6e9f198178298" gracePeriod=30 Oct 31 00:10:01.914818 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:01.914666 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="6885e4a3-3f18-477a-90ef-f59dbef4d404" containerName="tempo-gateway-opa" containerID="cri-o://db4bad30802608be7167243b935f595c692c98e9a2700be82e79a0e2173581af" gracePeriod=30 Oct 31 00:10:01.914818 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:01.914717 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="6885e4a3-3f18-477a-90ef-f59dbef4d404" containerName="tempo-gateway" containerID="cri-o://0999e1bfd89f852a24dd8bc572c374cce7bb0b18b8f7d475cd96a3cc548e1229" gracePeriod=30 Oct 31 00:10:01.923836 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:01.923793 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:02.003332 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.003299 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-thanos-prometheus-http-client-file\") pod \"6dd882e7-9623-45d4-96f1-0cd002fb7906\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " Oct 31 00:10:02.003514 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.003357 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dd882e7-9623-45d4-96f1-0cd002fb7906-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6dd882e7-9623-45d4-96f1-0cd002fb7906-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"6dd882e7-9623-45d4-96f1-0cd002fb7906\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " Oct 31 00:10:02.003514 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.003380 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"6dd882e7-9623-45d4-96f1-0cd002fb7906\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " Oct 31 00:10:02.003514 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.003405 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dd882e7-9623-45d4-96f1-0cd002fb7906-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6dd882e7-9623-45d4-96f1-0cd002fb7906-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"6dd882e7-9623-45d4-96f1-0cd002fb7906\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " Oct 31 00:10:02.003514 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.003494 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"6dd882e7-9623-45d4-96f1-0cd002fb7906\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " Oct 31 00:10:02.003735 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.003544 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-config\" (UniqueName: \"kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-config\") pod \"6dd882e7-9623-45d4-96f1-0cd002fb7906\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " Oct 31 00:10:02.003735 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.003582 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dd882e7-9623-45d4-96f1-0cd002fb7906-tls-assets\" (UniqueName: \"kubernetes.io/projected/6dd882e7-9623-45d4-96f1-0cd002fb7906-tls-assets\") pod \"6dd882e7-9623-45d4-96f1-0cd002fb7906\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " Oct 31 00:10:02.003735 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.003620 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-secret-prometheus-web-tls-ca\") pod \"6dd882e7-9623-45d4-96f1-0cd002fb7906\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " Oct 31 00:10:02.003735 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.003653 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dd882e7-9623-45d4-96f1-0cd002fb7906-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6dd882e7-9623-45d4-96f1-0cd002fb7906-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"6dd882e7-9623-45d4-96f1-0cd002fb7906\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " Oct 31 00:10:02.003735 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.003681 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dd882e7-9623-45d4-96f1-0cd002fb7906-kube-api-access-mnthd\" (UniqueName: \"kubernetes.io/projected/6dd882e7-9623-45d4-96f1-0cd002fb7906-kube-api-access-mnthd\") pod \"6dd882e7-9623-45d4-96f1-0cd002fb7906\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " Oct 31 00:10:02.003735 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.003710 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"6dd882e7-9623-45d4-96f1-0cd002fb7906\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " Oct 31 00:10:02.004080 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.003813 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-web-config\" (UniqueName: \"kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-web-config\") pod \"6dd882e7-9623-45d4-96f1-0cd002fb7906\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " Oct 31 00:10:02.004080 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.003867 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dd882e7-9623-45d4-96f1-0cd002fb7906-config-out\" (UniqueName: \"kubernetes.io/empty-dir/6dd882e7-9623-45d4-96f1-0cd002fb7906-config-out\") pod \"6dd882e7-9623-45d4-96f1-0cd002fb7906\" (UID: \"6dd882e7-9623-45d4-96f1-0cd002fb7906\") " Oct 31 00:10:02.004188 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.004155 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/406bd387-d4a8-4388-a6dd-cc38c86d36fd-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.004188 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.004175 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/8f0a400e-5e95-4403-af53-6108a54263ca-prometheus-client-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.004297 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.004192 5274 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f0a400e-5e95-4403-af53-6108a54263ca-kube-rbac-proxy-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.004297 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.004208 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdw77\" (UniqueName: \"kubernetes.io/projected/8f0a400e-5e95-4403-af53-6108a54263ca-kube-api-access-xdw77\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.004297 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.004222 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/8f0a400e-5e95-4403-af53-6108a54263ca-prometheus-ca\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.004297 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.004236 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/406bd387-d4a8-4388-a6dd-cc38c86d36fd-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.004297 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.004251 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/8f0a400e-5e95-4403-af53-6108a54263ca-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.004297 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.004264 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcbz9\" (UniqueName: \"kubernetes.io/projected/406bd387-d4a8-4388-a6dd-cc38c86d36fd-kube-api-access-tcbz9\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.004642 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.004322 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dd882e7-9623-45d4-96f1-0cd002fb7906-prometheus-data-science-monitoringstack-rulefiles-1" pod "6dd882e7-9623-45d4-96f1-0cd002fb7906" (UID: "6dd882e7-9623-45d4-96f1-0cd002fb7906"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:10:02.006127 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.005292 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dd882e7-9623-45d4-96f1-0cd002fb7906-prometheus-data-science-monitoringstack-rulefiles-2" pod "6dd882e7-9623-45d4-96f1-0cd002fb7906" (UID: "6dd882e7-9623-45d4-96f1-0cd002fb7906"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:10:02.007197 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.007155 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dd882e7-9623-45d4-96f1-0cd002fb7906-prometheus-data-science-monitoringstack-rulefiles-0" pod "6dd882e7-9623-45d4-96f1-0cd002fb7906" (UID: "6dd882e7-9623-45d4-96f1-0cd002fb7906"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:10:02.007805 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.007770 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dd882e7-9623-45d4-96f1-0cd002fb7906-config-out" pod "6dd882e7-9623-45d4-96f1-0cd002fb7906" (UID: "6dd882e7-9623-45d4-96f1-0cd002fb7906"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:02.008025 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.007999 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-thanos-prometheus-http-client-file" pod "6dd882e7-9623-45d4-96f1-0cd002fb7906" (UID: "6dd882e7-9623-45d4-96f1-0cd002fb7906"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:02.008843 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.008814 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "6dd882e7-9623-45d4-96f1-0cd002fb7906" (UID: "6dd882e7-9623-45d4-96f1-0cd002fb7906"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:02.010949 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.010889 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dd882e7-9623-45d4-96f1-0cd002fb7906-tls-assets" pod "6dd882e7-9623-45d4-96f1-0cd002fb7906" (UID: "6dd882e7-9623-45d4-96f1-0cd002fb7906"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:02.010949 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.010909 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-config" pod "6dd882e7-9623-45d4-96f1-0cd002fb7906" (UID: "6dd882e7-9623-45d4-96f1-0cd002fb7906"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:02.010949 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.011383 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "6dd882e7-9623-45d4-96f1-0cd002fb7906" (UID: "6dd882e7-9623-45d4-96f1-0cd002fb7906"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:02.013230 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.013198 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-secret-prometheus-web-tls-ca" pod "6dd882e7-9623-45d4-96f1-0cd002fb7906" (UID: "6dd882e7-9623-45d4-96f1-0cd002fb7906"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:02.013417 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.013392 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dd882e7-9623-45d4-96f1-0cd002fb7906-kube-api-access-mnthd" pod "6dd882e7-9623-45d4-96f1-0cd002fb7906" (UID: "6dd882e7-9623-45d4-96f1-0cd002fb7906"). InnerVolumeSpecName "kube-api-access-mnthd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:02.015409 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.015387 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a" pod "6dd882e7-9623-45d4-96f1-0cd002fb7906" (UID: "6dd882e7-9623-45d4-96f1-0cd002fb7906"). InnerVolumeSpecName "pvc-63e34709-8d85-4142-b53a-f655698606ff". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:10:02.037747 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.037720 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-web-config" pod "6dd882e7-9623-45d4-96f1-0cd002fb7906" (UID: "6dd882e7-9623-45d4-96f1-0cd002fb7906"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:02.105330 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.105291 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6dd882e7-9623-45d4-96f1-0cd002fb7906-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.105511 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.105341 5274 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" " Oct 31 00:10:02.105511 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.105352 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.105511 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.105361 5274 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6dd882e7-9623-45d4-96f1-0cd002fb7906-tls-assets\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.105511 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.105371 5274 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-secret-prometheus-web-tls-ca\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.105511 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.105380 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6dd882e7-9623-45d4-96f1-0cd002fb7906-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.105511 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.105389 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnthd\" (UniqueName: \"kubernetes.io/projected/6dd882e7-9623-45d4-96f1-0cd002fb7906-kube-api-access-mnthd\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.105511 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.105401 5274 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.105511 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.105419 5274 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-web-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.105511 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.105430 5274 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6dd882e7-9623-45d4-96f1-0cd002fb7906-config-out\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.105511 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.105439 5274 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.105511 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.105448 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6dd882e7-9623-45d4-96f1-0cd002fb7906-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.105511 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.105457 5274 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6dd882e7-9623-45d4-96f1-0cd002fb7906-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.118947 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.118925 5274 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-63e34709-8d85-4142-b53a-f655698606ff" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a") on node "ip-10-0-129-70.us-east-2.compute.internal" Oct 31 00:10:02.206759 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.206692 5274 reconciler_common.go:299] "Volume detached for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"csi-3beb91b3ebf177571ca474589da3560cd18787cfa50b2a0b9bfef85661e6cf21\"" Oct 31 00:10:02.319347 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.319301 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="c14cc7ba-1add-4829-806a-639de0bc93eb" containerName="alertmanager" probeResult="failure" output="Get \"http://10.134.0.45:9093/-/ready\": dial tcp 10.134.0.45:9093: connect: connection refused" Oct 31 00:10:02.427683 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.427660 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:10:02.509495 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.509429 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c14cc7ba-1add-4829-806a-639de0bc93eb-web-config\" (UniqueName: \"kubernetes.io/secret/c14cc7ba-1add-4829-806a-639de0bc93eb-web-config\") pod \"c14cc7ba-1add-4829-806a-639de0bc93eb\" (UID: \"c14cc7ba-1add-4829-806a-639de0bc93eb\") " Oct 31 00:10:02.509495 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.509468 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c14cc7ba-1add-4829-806a-639de0bc93eb-tls-assets\" (UniqueName: \"kubernetes.io/projected/c14cc7ba-1add-4829-806a-639de0bc93eb-tls-assets\") pod \"c14cc7ba-1add-4829-806a-639de0bc93eb\" (UID: \"c14cc7ba-1add-4829-806a-639de0bc93eb\") " Oct 31 00:10:02.509675 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.509521 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c14cc7ba-1add-4829-806a-639de0bc93eb-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c14cc7ba-1add-4829-806a-639de0bc93eb-cluster-tls-config\") pod \"c14cc7ba-1add-4829-806a-639de0bc93eb\" (UID: \"c14cc7ba-1add-4829-806a-639de0bc93eb\") " Oct 31 00:10:02.509675 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.509562 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c14cc7ba-1add-4829-806a-639de0bc93eb-config-volume\" (UniqueName: \"kubernetes.io/secret/c14cc7ba-1add-4829-806a-639de0bc93eb-config-volume\") pod \"c14cc7ba-1add-4829-806a-639de0bc93eb\" (UID: \"c14cc7ba-1add-4829-806a-639de0bc93eb\") " Oct 31 00:10:02.509675 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.509600 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c14cc7ba-1add-4829-806a-639de0bc93eb-config-out\" (UniqueName: \"kubernetes.io/empty-dir/c14cc7ba-1add-4829-806a-639de0bc93eb-config-out\") pod \"c14cc7ba-1add-4829-806a-639de0bc93eb\" (UID: \"c14cc7ba-1add-4829-806a-639de0bc93eb\") " Oct 31 00:10:02.509675 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.509659 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c14cc7ba-1add-4829-806a-639de0bc93eb-kube-api-access-ljddk\" (UniqueName: \"kubernetes.io/projected/c14cc7ba-1add-4829-806a-639de0bc93eb-kube-api-access-ljddk\") pod \"c14cc7ba-1add-4829-806a-639de0bc93eb\" (UID: \"c14cc7ba-1add-4829-806a-639de0bc93eb\") " Oct 31 00:10:02.510112 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.509698 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c14cc7ba-1add-4829-806a-639de0bc93eb-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/c14cc7ba-1add-4829-806a-639de0bc93eb-alertmanager-data-science-monitoringstack-db\") pod \"c14cc7ba-1add-4829-806a-639de0bc93eb\" (UID: \"c14cc7ba-1add-4829-806a-639de0bc93eb\") " Oct 31 00:10:02.510181 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.510120 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c14cc7ba-1add-4829-806a-639de0bc93eb-alertmanager-data-science-monitoringstack-db" pod "c14cc7ba-1add-4829-806a-639de0bc93eb" (UID: "c14cc7ba-1add-4829-806a-639de0bc93eb"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:02.512106 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.512044 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c14cc7ba-1add-4829-806a-639de0bc93eb-config-volume" pod "c14cc7ba-1add-4829-806a-639de0bc93eb" (UID: "c14cc7ba-1add-4829-806a-639de0bc93eb"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:02.512106 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.512047 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c14cc7ba-1add-4829-806a-639de0bc93eb-config-out" pod "c14cc7ba-1add-4829-806a-639de0bc93eb" (UID: "c14cc7ba-1add-4829-806a-639de0bc93eb"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:02.512290 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.512144 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c14cc7ba-1add-4829-806a-639de0bc93eb-tls-assets" pod "c14cc7ba-1add-4829-806a-639de0bc93eb" (UID: "c14cc7ba-1add-4829-806a-639de0bc93eb"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:02.512290 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.512197 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c14cc7ba-1add-4829-806a-639de0bc93eb-kube-api-access-ljddk" pod "c14cc7ba-1add-4829-806a-639de0bc93eb" (UID: "c14cc7ba-1add-4829-806a-639de0bc93eb"). InnerVolumeSpecName "kube-api-access-ljddk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:02.515933 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.515905 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c14cc7ba-1add-4829-806a-639de0bc93eb-cluster-tls-config" pod "c14cc7ba-1add-4829-806a-639de0bc93eb" (UID: "c14cc7ba-1add-4829-806a-639de0bc93eb"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:02.522742 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.522719 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c14cc7ba-1add-4829-806a-639de0bc93eb-web-config" pod "c14cc7ba-1add-4829-806a-639de0bc93eb" (UID: "c14cc7ba-1add-4829-806a-639de0bc93eb"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:02.535301 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.535284 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2plj2" Oct 31 00:10:02.535381 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.535303 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-2plj2" event={"ID":"406bd387-d4a8-4388-a6dd-cc38c86d36fd","Type":"ContainerDied","Data":"bdf366ad3a66d877c041f14ebe46cb1458a8cf7a01cf20314cbee795f88e3b2a"} Oct 31 00:10:02.535381 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.535359 5274 scope.go:122] "RemoveContainer" containerID="3cae132b592b809f49d189e7c23d221d1d3019c4beaccad5817fd8db1d59cb16" Oct 31 00:10:02.537920 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.537891 5274 generic.go:356] "Generic (PLEG): container finished" podID="6dd882e7-9623-45d4-96f1-0cd002fb7906" containerID="999e2aaccd33e5183a4c655b8ce6ab5eceab442612f98fae15c8fa7e8e8268a9" exitCode=0 Oct 31 00:10:02.538034 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.537922 5274 generic.go:356] "Generic (PLEG): container finished" podID="6dd882e7-9623-45d4-96f1-0cd002fb7906" containerID="6bdde96d361cbd4ea3cb2cca44731a726743511d45dbf6ad661524b43ffcdb14" exitCode=0 Oct 31 00:10:02.538034 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.537930 5274 generic.go:356] "Generic (PLEG): container finished" podID="6dd882e7-9623-45d4-96f1-0cd002fb7906" containerID="5ffa85c5e5daa8106bc8ea56a7939efa6abdda48e89272ea6b4ced9f898fcc18" exitCode=0 Oct 31 00:10:02.538034 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.537979 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:10:02.538034 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.537989 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"6dd882e7-9623-45d4-96f1-0cd002fb7906","Type":"ContainerDied","Data":"999e2aaccd33e5183a4c655b8ce6ab5eceab442612f98fae15c8fa7e8e8268a9"} Oct 31 00:10:02.538242 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.538052 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"6dd882e7-9623-45d4-96f1-0cd002fb7906","Type":"ContainerDied","Data":"6bdde96d361cbd4ea3cb2cca44731a726743511d45dbf6ad661524b43ffcdb14"} Oct 31 00:10:02.538242 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.538080 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"6dd882e7-9623-45d4-96f1-0cd002fb7906","Type":"ContainerDied","Data":"5ffa85c5e5daa8106bc8ea56a7939efa6abdda48e89272ea6b4ced9f898fcc18"} Oct 31 00:10:02.538242 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.538110 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"6dd882e7-9623-45d4-96f1-0cd002fb7906","Type":"ContainerDied","Data":"147dc4175ab1531e540032011e51ccef7f164da31f2b49fcaefd1843776ac519"} Oct 31 00:10:02.540553 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.540530 5274 generic.go:356] "Generic (PLEG): container finished" podID="c14cc7ba-1add-4829-806a-639de0bc93eb" containerID="4dc0fed8214bf715d83257e4c8107ab14e73f84c779bb28d9a64c2319670e0b7" exitCode=0 Oct 31 00:10:02.540639 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.540555 5274 generic.go:356] "Generic (PLEG): container finished" podID="c14cc7ba-1add-4829-806a-639de0bc93eb" containerID="3adcf308fd53cf1fa56b17d3c6935c8c0ddc1b5b21b5f6e6c0ed6bf6ee17ae3c" exitCode=0 Oct 31 00:10:02.540639 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.540561 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:10:02.540639 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.540595 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"c14cc7ba-1add-4829-806a-639de0bc93eb","Type":"ContainerDied","Data":"4dc0fed8214bf715d83257e4c8107ab14e73f84c779bb28d9a64c2319670e0b7"} Oct 31 00:10:02.540836 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.540650 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"c14cc7ba-1add-4829-806a-639de0bc93eb","Type":"ContainerDied","Data":"3adcf308fd53cf1fa56b17d3c6935c8c0ddc1b5b21b5f6e6c0ed6bf6ee17ae3c"} Oct 31 00:10:02.540836 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.540687 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"c14cc7ba-1add-4829-806a-639de0bc93eb","Type":"ContainerDied","Data":"3b78481c14a5ff6f1d214e0152c2de901e34c033c2efda66165873ddcfb7e509"} Oct 31 00:10:02.543816 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.543794 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m2s4f" event={"ID":"8f0a400e-5e95-4403-af53-6108a54263ca","Type":"ContainerDied","Data":"53635424bc47a858c1058335782ad6faae6cfdc57eed40704ab69b0f84a5e171"} Oct 31 00:10:02.543912 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.543874 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m2s4f" Oct 31 00:10:02.545799 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.545778 5274 scope.go:122] "RemoveContainer" containerID="999e2aaccd33e5183a4c655b8ce6ab5eceab442612f98fae15c8fa7e8e8268a9" Oct 31 00:10:02.547547 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.547521 5274 generic.go:356] "Generic (PLEG): container finished" podID="6885e4a3-3f18-477a-90ef-f59dbef4d404" containerID="db4bad30802608be7167243b935f595c692c98e9a2700be82e79a0e2173581af" exitCode=0 Oct 31 00:10:02.547547 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.547546 5274 generic.go:356] "Generic (PLEG): container finished" podID="6885e4a3-3f18-477a-90ef-f59dbef4d404" containerID="eff1b373ae1994af65d6c6cdde9b0a7c7fd4a2524c98d0c10dd6e9f198178298" exitCode=0 Oct 31 00:10:02.547702 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.547605 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"6885e4a3-3f18-477a-90ef-f59dbef4d404","Type":"ContainerDied","Data":"db4bad30802608be7167243b935f595c692c98e9a2700be82e79a0e2173581af"} Oct 31 00:10:02.547702 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.547658 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"6885e4a3-3f18-477a-90ef-f59dbef4d404","Type":"ContainerDied","Data":"eff1b373ae1994af65d6c6cdde9b0a7c7fd4a2524c98d0c10dd6e9f198178298"} Oct 31 00:10:02.554779 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.554761 5274 scope.go:122] "RemoveContainer" containerID="6bdde96d361cbd4ea3cb2cca44731a726743511d45dbf6ad661524b43ffcdb14" Oct 31 00:10:02.562985 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.562941 5274 scope.go:122] "RemoveContainer" containerID="5ffa85c5e5daa8106bc8ea56a7939efa6abdda48e89272ea6b4ced9f898fcc18" Oct 31 00:10:02.566563 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.566539 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-2plj2"] Oct 31 00:10:02.571087 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.571068 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-2plj2"] Oct 31 00:10:02.571357 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.571334 5274 scope.go:122] "RemoveContainer" containerID="9ee773905955211ce11f090799d5253828696f1b1b383b5adffbb8e23f3faca0" Oct 31 00:10:02.579040 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.579022 5274 scope.go:122] "RemoveContainer" containerID="999e2aaccd33e5183a4c655b8ce6ab5eceab442612f98fae15c8fa7e8e8268a9" Oct 31 00:10:02.579285 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:10:02.579267 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"999e2aaccd33e5183a4c655b8ce6ab5eceab442612f98fae15c8fa7e8e8268a9\": container with ID starting with 999e2aaccd33e5183a4c655b8ce6ab5eceab442612f98fae15c8fa7e8e8268a9 not found: ID does not exist" containerID="999e2aaccd33e5183a4c655b8ce6ab5eceab442612f98fae15c8fa7e8e8268a9" Oct 31 00:10:02.579335 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.579292 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"999e2aaccd33e5183a4c655b8ce6ab5eceab442612f98fae15c8fa7e8e8268a9"} err="failed to get container status \"999e2aaccd33e5183a4c655b8ce6ab5eceab442612f98fae15c8fa7e8e8268a9\": rpc error: code = NotFound desc = could not find container \"999e2aaccd33e5183a4c655b8ce6ab5eceab442612f98fae15c8fa7e8e8268a9\": container with ID starting with 999e2aaccd33e5183a4c655b8ce6ab5eceab442612f98fae15c8fa7e8e8268a9 not found: ID does not exist" Oct 31 00:10:02.579335 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.579308 5274 scope.go:122] "RemoveContainer" containerID="6bdde96d361cbd4ea3cb2cca44731a726743511d45dbf6ad661524b43ffcdb14" Oct 31 00:10:02.579536 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:10:02.579521 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bdde96d361cbd4ea3cb2cca44731a726743511d45dbf6ad661524b43ffcdb14\": container with ID starting with 6bdde96d361cbd4ea3cb2cca44731a726743511d45dbf6ad661524b43ffcdb14 not found: ID does not exist" containerID="6bdde96d361cbd4ea3cb2cca44731a726743511d45dbf6ad661524b43ffcdb14" Oct 31 00:10:02.579572 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.579540 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bdde96d361cbd4ea3cb2cca44731a726743511d45dbf6ad661524b43ffcdb14"} err="failed to get container status \"6bdde96d361cbd4ea3cb2cca44731a726743511d45dbf6ad661524b43ffcdb14\": rpc error: code = NotFound desc = could not find container \"6bdde96d361cbd4ea3cb2cca44731a726743511d45dbf6ad661524b43ffcdb14\": container with ID starting with 6bdde96d361cbd4ea3cb2cca44731a726743511d45dbf6ad661524b43ffcdb14 not found: ID does not exist" Oct 31 00:10:02.579572 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.579552 5274 scope.go:122] "RemoveContainer" containerID="5ffa85c5e5daa8106bc8ea56a7939efa6abdda48e89272ea6b4ced9f898fcc18" Oct 31 00:10:02.579743 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:10:02.579727 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ffa85c5e5daa8106bc8ea56a7939efa6abdda48e89272ea6b4ced9f898fcc18\": container with ID starting with 5ffa85c5e5daa8106bc8ea56a7939efa6abdda48e89272ea6b4ced9f898fcc18 not found: ID does not exist" containerID="5ffa85c5e5daa8106bc8ea56a7939efa6abdda48e89272ea6b4ced9f898fcc18" Oct 31 00:10:02.579781 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.579748 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ffa85c5e5daa8106bc8ea56a7939efa6abdda48e89272ea6b4ced9f898fcc18"} err="failed to get container status \"5ffa85c5e5daa8106bc8ea56a7939efa6abdda48e89272ea6b4ced9f898fcc18\": rpc error: code = NotFound desc = could not find container \"5ffa85c5e5daa8106bc8ea56a7939efa6abdda48e89272ea6b4ced9f898fcc18\": container with ID starting with 5ffa85c5e5daa8106bc8ea56a7939efa6abdda48e89272ea6b4ced9f898fcc18 not found: ID does not exist" Oct 31 00:10:02.579781 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.579765 5274 scope.go:122] "RemoveContainer" containerID="9ee773905955211ce11f090799d5253828696f1b1b383b5adffbb8e23f3faca0" Oct 31 00:10:02.579950 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:10:02.579936 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ee773905955211ce11f090799d5253828696f1b1b383b5adffbb8e23f3faca0\": container with ID starting with 9ee773905955211ce11f090799d5253828696f1b1b383b5adffbb8e23f3faca0 not found: ID does not exist" containerID="9ee773905955211ce11f090799d5253828696f1b1b383b5adffbb8e23f3faca0" Oct 31 00:10:02.580024 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.580009 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ee773905955211ce11f090799d5253828696f1b1b383b5adffbb8e23f3faca0"} err="failed to get container status \"9ee773905955211ce11f090799d5253828696f1b1b383b5adffbb8e23f3faca0\": rpc error: code = NotFound desc = could not find container \"9ee773905955211ce11f090799d5253828696f1b1b383b5adffbb8e23f3faca0\": container with ID starting with 9ee773905955211ce11f090799d5253828696f1b1b383b5adffbb8e23f3faca0 not found: ID does not exist" Oct 31 00:10:02.580059 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.580025 5274 scope.go:122] "RemoveContainer" containerID="999e2aaccd33e5183a4c655b8ce6ab5eceab442612f98fae15c8fa7e8e8268a9" Oct 31 00:10:02.580235 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.580218 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"999e2aaccd33e5183a4c655b8ce6ab5eceab442612f98fae15c8fa7e8e8268a9"} err="failed to get container status \"999e2aaccd33e5183a4c655b8ce6ab5eceab442612f98fae15c8fa7e8e8268a9\": rpc error: code = NotFound desc = could not find container \"999e2aaccd33e5183a4c655b8ce6ab5eceab442612f98fae15c8fa7e8e8268a9\": container with ID starting with 999e2aaccd33e5183a4c655b8ce6ab5eceab442612f98fae15c8fa7e8e8268a9 not found: ID does not exist" Oct 31 00:10:02.580278 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.580236 5274 scope.go:122] "RemoveContainer" containerID="6bdde96d361cbd4ea3cb2cca44731a726743511d45dbf6ad661524b43ffcdb14" Oct 31 00:10:02.580433 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.580417 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bdde96d361cbd4ea3cb2cca44731a726743511d45dbf6ad661524b43ffcdb14"} err="failed to get container status \"6bdde96d361cbd4ea3cb2cca44731a726743511d45dbf6ad661524b43ffcdb14\": rpc error: code = NotFound desc = could not find container \"6bdde96d361cbd4ea3cb2cca44731a726743511d45dbf6ad661524b43ffcdb14\": container with ID starting with 6bdde96d361cbd4ea3cb2cca44731a726743511d45dbf6ad661524b43ffcdb14 not found: ID does not exist" Oct 31 00:10:02.580473 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.580433 5274 scope.go:122] "RemoveContainer" containerID="5ffa85c5e5daa8106bc8ea56a7939efa6abdda48e89272ea6b4ced9f898fcc18" Oct 31 00:10:02.580604 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.580590 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ffa85c5e5daa8106bc8ea56a7939efa6abdda48e89272ea6b4ced9f898fcc18"} err="failed to get container status \"5ffa85c5e5daa8106bc8ea56a7939efa6abdda48e89272ea6b4ced9f898fcc18\": rpc error: code = NotFound desc = could not find container \"5ffa85c5e5daa8106bc8ea56a7939efa6abdda48e89272ea6b4ced9f898fcc18\": container with ID starting with 5ffa85c5e5daa8106bc8ea56a7939efa6abdda48e89272ea6b4ced9f898fcc18 not found: ID does not exist" Oct 31 00:10:02.580645 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.580604 5274 scope.go:122] "RemoveContainer" containerID="9ee773905955211ce11f090799d5253828696f1b1b383b5adffbb8e23f3faca0" Oct 31 00:10:02.580773 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.580756 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ee773905955211ce11f090799d5253828696f1b1b383b5adffbb8e23f3faca0"} err="failed to get container status \"9ee773905955211ce11f090799d5253828696f1b1b383b5adffbb8e23f3faca0\": rpc error: code = NotFound desc = could not find container \"9ee773905955211ce11f090799d5253828696f1b1b383b5adffbb8e23f3faca0\": container with ID starting with 9ee773905955211ce11f090799d5253828696f1b1b383b5adffbb8e23f3faca0 not found: ID does not exist" Oct 31 00:10:02.580812 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.580778 5274 scope.go:122] "RemoveContainer" containerID="999e2aaccd33e5183a4c655b8ce6ab5eceab442612f98fae15c8fa7e8e8268a9" Oct 31 00:10:02.580950 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.580934 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"999e2aaccd33e5183a4c655b8ce6ab5eceab442612f98fae15c8fa7e8e8268a9"} err="failed to get container status \"999e2aaccd33e5183a4c655b8ce6ab5eceab442612f98fae15c8fa7e8e8268a9\": rpc error: code = NotFound desc = could not find container \"999e2aaccd33e5183a4c655b8ce6ab5eceab442612f98fae15c8fa7e8e8268a9\": container with ID starting with 999e2aaccd33e5183a4c655b8ce6ab5eceab442612f98fae15c8fa7e8e8268a9 not found: ID does not exist" Oct 31 00:10:02.580950 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.580962 5274 scope.go:122] "RemoveContainer" containerID="6bdde96d361cbd4ea3cb2cca44731a726743511d45dbf6ad661524b43ffcdb14" Oct 31 00:10:02.581210 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.581193 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bdde96d361cbd4ea3cb2cca44731a726743511d45dbf6ad661524b43ffcdb14"} err="failed to get container status \"6bdde96d361cbd4ea3cb2cca44731a726743511d45dbf6ad661524b43ffcdb14\": rpc error: code = NotFound desc = could not find container \"6bdde96d361cbd4ea3cb2cca44731a726743511d45dbf6ad661524b43ffcdb14\": container with ID starting with 6bdde96d361cbd4ea3cb2cca44731a726743511d45dbf6ad661524b43ffcdb14 not found: ID does not exist" Oct 31 00:10:02.581210 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.581209 5274 scope.go:122] "RemoveContainer" containerID="5ffa85c5e5daa8106bc8ea56a7939efa6abdda48e89272ea6b4ced9f898fcc18" Oct 31 00:10:02.581447 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.581427 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ffa85c5e5daa8106bc8ea56a7939efa6abdda48e89272ea6b4ced9f898fcc18"} err="failed to get container status \"5ffa85c5e5daa8106bc8ea56a7939efa6abdda48e89272ea6b4ced9f898fcc18\": rpc error: code = NotFound desc = could not find container \"5ffa85c5e5daa8106bc8ea56a7939efa6abdda48e89272ea6b4ced9f898fcc18\": container with ID starting with 5ffa85c5e5daa8106bc8ea56a7939efa6abdda48e89272ea6b4ced9f898fcc18 not found: ID does not exist" Oct 31 00:10:02.581509 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.581448 5274 scope.go:122] "RemoveContainer" containerID="9ee773905955211ce11f090799d5253828696f1b1b383b5adffbb8e23f3faca0" Oct 31 00:10:02.581679 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.581664 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ee773905955211ce11f090799d5253828696f1b1b383b5adffbb8e23f3faca0"} err="failed to get container status \"9ee773905955211ce11f090799d5253828696f1b1b383b5adffbb8e23f3faca0\": rpc error: code = NotFound desc = could not find container \"9ee773905955211ce11f090799d5253828696f1b1b383b5adffbb8e23f3faca0\": container with ID starting with 9ee773905955211ce11f090799d5253828696f1b1b383b5adffbb8e23f3faca0 not found: ID does not exist" Oct 31 00:10:02.581679 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.581679 5274 scope.go:122] "RemoveContainer" containerID="4dc0fed8214bf715d83257e4c8107ab14e73f84c779bb28d9a64c2319670e0b7" Oct 31 00:10:02.583905 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.583886 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:10:02.589206 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.589188 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:10:02.591211 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.591190 5274 scope.go:122] "RemoveContainer" containerID="3adcf308fd53cf1fa56b17d3c6935c8c0ddc1b5b21b5f6e6c0ed6bf6ee17ae3c" Oct 31 00:10:02.598519 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.598502 5274 scope.go:122] "RemoveContainer" containerID="b65cd7e92f819ee6860d78f0d097f3d94a34f86f9f694281d5dd8c91fc0e812d" Oct 31 00:10:02.602992 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.602975 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:10:02.606033 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.606017 5274 scope.go:122] "RemoveContainer" containerID="4dc0fed8214bf715d83257e4c8107ab14e73f84c779bb28d9a64c2319670e0b7" Oct 31 00:10:02.606287 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:10:02.606269 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dc0fed8214bf715d83257e4c8107ab14e73f84c779bb28d9a64c2319670e0b7\": container with ID starting with 4dc0fed8214bf715d83257e4c8107ab14e73f84c779bb28d9a64c2319670e0b7 not found: ID does not exist" containerID="4dc0fed8214bf715d83257e4c8107ab14e73f84c779bb28d9a64c2319670e0b7" Oct 31 00:10:02.606344 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.606293 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dc0fed8214bf715d83257e4c8107ab14e73f84c779bb28d9a64c2319670e0b7"} err="failed to get container status \"4dc0fed8214bf715d83257e4c8107ab14e73f84c779bb28d9a64c2319670e0b7\": rpc error: code = NotFound desc = could not find container \"4dc0fed8214bf715d83257e4c8107ab14e73f84c779bb28d9a64c2319670e0b7\": container with ID starting with 4dc0fed8214bf715d83257e4c8107ab14e73f84c779bb28d9a64c2319670e0b7 not found: ID does not exist" Oct 31 00:10:02.606344 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.606306 5274 scope.go:122] "RemoveContainer" containerID="3adcf308fd53cf1fa56b17d3c6935c8c0ddc1b5b21b5f6e6c0ed6bf6ee17ae3c" Oct 31 00:10:02.606524 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:10:02.606508 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3adcf308fd53cf1fa56b17d3c6935c8c0ddc1b5b21b5f6e6c0ed6bf6ee17ae3c\": container with ID starting with 3adcf308fd53cf1fa56b17d3c6935c8c0ddc1b5b21b5f6e6c0ed6bf6ee17ae3c not found: ID does not exist" containerID="3adcf308fd53cf1fa56b17d3c6935c8c0ddc1b5b21b5f6e6c0ed6bf6ee17ae3c" Oct 31 00:10:02.606560 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.606527 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3adcf308fd53cf1fa56b17d3c6935c8c0ddc1b5b21b5f6e6c0ed6bf6ee17ae3c"} err="failed to get container status \"3adcf308fd53cf1fa56b17d3c6935c8c0ddc1b5b21b5f6e6c0ed6bf6ee17ae3c\": rpc error: code = NotFound desc = could not find container \"3adcf308fd53cf1fa56b17d3c6935c8c0ddc1b5b21b5f6e6c0ed6bf6ee17ae3c\": container with ID starting with 3adcf308fd53cf1fa56b17d3c6935c8c0ddc1b5b21b5f6e6c0ed6bf6ee17ae3c not found: ID does not exist" Oct 31 00:10:02.606560 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.606540 5274 scope.go:122] "RemoveContainer" containerID="b65cd7e92f819ee6860d78f0d097f3d94a34f86f9f694281d5dd8c91fc0e812d" Oct 31 00:10:02.606774 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:10:02.606756 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b65cd7e92f819ee6860d78f0d097f3d94a34f86f9f694281d5dd8c91fc0e812d\": container with ID starting with b65cd7e92f819ee6860d78f0d097f3d94a34f86f9f694281d5dd8c91fc0e812d not found: ID does not exist" containerID="b65cd7e92f819ee6860d78f0d097f3d94a34f86f9f694281d5dd8c91fc0e812d" Oct 31 00:10:02.606814 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.606779 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b65cd7e92f819ee6860d78f0d097f3d94a34f86f9f694281d5dd8c91fc0e812d"} err="failed to get container status \"b65cd7e92f819ee6860d78f0d097f3d94a34f86f9f694281d5dd8c91fc0e812d\": rpc error: code = NotFound desc = could not find container \"b65cd7e92f819ee6860d78f0d097f3d94a34f86f9f694281d5dd8c91fc0e812d\": container with ID starting with b65cd7e92f819ee6860d78f0d097f3d94a34f86f9f694281d5dd8c91fc0e812d not found: ID does not exist" Oct 31 00:10:02.606814 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.606793 5274 scope.go:122] "RemoveContainer" containerID="4dc0fed8214bf715d83257e4c8107ab14e73f84c779bb28d9a64c2319670e0b7" Oct 31 00:10:02.607026 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.607006 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dc0fed8214bf715d83257e4c8107ab14e73f84c779bb28d9a64c2319670e0b7"} err="failed to get container status \"4dc0fed8214bf715d83257e4c8107ab14e73f84c779bb28d9a64c2319670e0b7\": rpc error: code = NotFound desc = could not find container \"4dc0fed8214bf715d83257e4c8107ab14e73f84c779bb28d9a64c2319670e0b7\": container with ID starting with 4dc0fed8214bf715d83257e4c8107ab14e73f84c779bb28d9a64c2319670e0b7 not found: ID does not exist" Oct 31 00:10:02.607084 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.607028 5274 scope.go:122] "RemoveContainer" containerID="3adcf308fd53cf1fa56b17d3c6935c8c0ddc1b5b21b5f6e6c0ed6bf6ee17ae3c" Oct 31 00:10:02.607278 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.607260 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3adcf308fd53cf1fa56b17d3c6935c8c0ddc1b5b21b5f6e6c0ed6bf6ee17ae3c"} err="failed to get container status \"3adcf308fd53cf1fa56b17d3c6935c8c0ddc1b5b21b5f6e6c0ed6bf6ee17ae3c\": rpc error: code = NotFound desc = could not find container \"3adcf308fd53cf1fa56b17d3c6935c8c0ddc1b5b21b5f6e6c0ed6bf6ee17ae3c\": container with ID starting with 3adcf308fd53cf1fa56b17d3c6935c8c0ddc1b5b21b5f6e6c0ed6bf6ee17ae3c not found: ID does not exist" Oct 31 00:10:02.607394 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.607279 5274 scope.go:122] "RemoveContainer" containerID="b65cd7e92f819ee6860d78f0d097f3d94a34f86f9f694281d5dd8c91fc0e812d" Oct 31 00:10:02.607500 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.607478 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b65cd7e92f819ee6860d78f0d097f3d94a34f86f9f694281d5dd8c91fc0e812d"} err="failed to get container status \"b65cd7e92f819ee6860d78f0d097f3d94a34f86f9f694281d5dd8c91fc0e812d\": rpc error: code = NotFound desc = could not find container \"b65cd7e92f819ee6860d78f0d097f3d94a34f86f9f694281d5dd8c91fc0e812d\": container with ID starting with b65cd7e92f819ee6860d78f0d097f3d94a34f86f9f694281d5dd8c91fc0e812d not found: ID does not exist" Oct 31 00:10:02.607558 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.607501 5274 scope.go:122] "RemoveContainer" containerID="30e2112ae4ba567c5e78dc7b967fea51e8c179ea7a7d52b7089711469e86f8ae" Oct 31 00:10:02.609843 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.609827 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:10:02.611051 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.611030 5274 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c14cc7ba-1add-4829-806a-639de0bc93eb-config-volume\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.611144 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.611055 5274 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c14cc7ba-1add-4829-806a-639de0bc93eb-config-out\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.611144 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.611072 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljddk\" (UniqueName: \"kubernetes.io/projected/c14cc7ba-1add-4829-806a-639de0bc93eb-kube-api-access-ljddk\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.611144 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.611088 5274 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/c14cc7ba-1add-4829-806a-639de0bc93eb-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.611144 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.611105 5274 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c14cc7ba-1add-4829-806a-639de0bc93eb-web-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.611144 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.611121 5274 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c14cc7ba-1add-4829-806a-639de0bc93eb-tls-assets\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.611346 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.611182 5274 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c14cc7ba-1add-4829-806a-639de0bc93eb-cluster-tls-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.621523 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.621505 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m2s4f"] Oct 31 00:10:02.626279 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:02.626263 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-m2s4f"] Oct 31 00:10:03.141597 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.141574 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:10:03.318311 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.318244 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6885e4a3-3f18-477a-90ef-f59dbef4d404-kube-api-access-qfm2t\" (UniqueName: \"kubernetes.io/projected/6885e4a3-3f18-477a-90ef-f59dbef4d404-kube-api-access-qfm2t\") pod \"6885e4a3-3f18-477a-90ef-f59dbef4d404\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") " Oct 31 00:10:03.318311 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.318275 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6885e4a3-3f18-477a-90ef-f59dbef4d404-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/6885e4a3-3f18-477a-90ef-f59dbef4d404-tempo-data-science-tempomonolithic-mtls\") pod \"6885e4a3-3f18-477a-90ef-f59dbef4d404\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") " Oct 31 00:10:03.318498 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.318351 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"6885e4a3-3f18-477a-90ef-f59dbef4d404\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") " Oct 31 00:10:03.318498 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.318380 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6885e4a3-3f18-477a-90ef-f59dbef4d404-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/6885e4a3-3f18-477a-90ef-f59dbef4d404-tempo-conf\") pod \"6885e4a3-3f18-477a-90ef-f59dbef4d404\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") " Oct 31 00:10:03.318498 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.318421 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6885e4a3-3f18-477a-90ef-f59dbef4d404-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/6885e4a3-3f18-477a-90ef-f59dbef4d404-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"6885e4a3-3f18-477a-90ef-f59dbef4d404\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") " Oct 31 00:10:03.318498 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.318458 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6885e4a3-3f18-477a-90ef-f59dbef4d404-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6885e4a3-3f18-477a-90ef-f59dbef4d404-tempo-data-science-tempomonolithic-ca-bundle\") pod \"6885e4a3-3f18-477a-90ef-f59dbef4d404\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") " Oct 31 00:10:03.318498 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.318475 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6885e4a3-3f18-477a-90ef-f59dbef4d404-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/6885e4a3-3f18-477a-90ef-f59dbef4d404-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"6885e4a3-3f18-477a-90ef-f59dbef4d404\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") " Oct 31 00:10:03.318498 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.318495 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6885e4a3-3f18-477a-90ef-f59dbef4d404-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/6885e4a3-3f18-477a-90ef-f59dbef4d404-gateway-rbac\") pod \"6885e4a3-3f18-477a-90ef-f59dbef4d404\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") " Oct 31 00:10:03.318804 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.318544 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6885e4a3-3f18-477a-90ef-f59dbef4d404-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/6885e4a3-3f18-477a-90ef-f59dbef4d404-gateway-tenants\") pod \"6885e4a3-3f18-477a-90ef-f59dbef4d404\" (UID: \"6885e4a3-3f18-477a-90ef-f59dbef4d404\") " Oct 31 00:10:03.318863 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.318827 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6885e4a3-3f18-477a-90ef-f59dbef4d404-tempo-data-science-tempomonolithic-serving-cabundle" pod "6885e4a3-3f18-477a-90ef-f59dbef4d404" (UID: "6885e4a3-3f18-477a-90ef-f59dbef4d404"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:10:03.319171 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.319140 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6885e4a3-3f18-477a-90ef-f59dbef4d404-tempo-conf" pod "6885e4a3-3f18-477a-90ef-f59dbef4d404" (UID: "6885e4a3-3f18-477a-90ef-f59dbef4d404"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:10:03.319171 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.319153 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6885e4a3-3f18-477a-90ef-f59dbef4d404-tempo-data-science-tempomonolithic-ca-bundle" pod "6885e4a3-3f18-477a-90ef-f59dbef4d404" (UID: "6885e4a3-3f18-477a-90ef-f59dbef4d404"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:10:03.319362 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.319179 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6885e4a3-3f18-477a-90ef-f59dbef4d404-gateway-rbac" pod "6885e4a3-3f18-477a-90ef-f59dbef4d404" (UID: "6885e4a3-3f18-477a-90ef-f59dbef4d404"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:10:03.320655 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.320620 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6885e4a3-3f18-477a-90ef-f59dbef4d404-kube-api-access-qfm2t" pod "6885e4a3-3f18-477a-90ef-f59dbef4d404" (UID: "6885e4a3-3f18-477a-90ef-f59dbef4d404"). InnerVolumeSpecName "kube-api-access-qfm2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:03.320777 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.320698 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6885e4a3-3f18-477a-90ef-f59dbef4d404-gateway-tenants" pod "6885e4a3-3f18-477a-90ef-f59dbef4d404" (UID: "6885e4a3-3f18-477a-90ef-f59dbef4d404"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:03.320861 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.320828 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6885e4a3-3f18-477a-90ef-f59dbef4d404-tempo-data-science-tempomonolithic-mtls" pod "6885e4a3-3f18-477a-90ef-f59dbef4d404" (UID: "6885e4a3-3f18-477a-90ef-f59dbef4d404"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:03.321164 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.321140 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6885e4a3-3f18-477a-90ef-f59dbef4d404-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "6885e4a3-3f18-477a-90ef-f59dbef4d404" (UID: "6885e4a3-3f18-477a-90ef-f59dbef4d404"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:03.321267 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.321249 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062" pod "6885e4a3-3f18-477a-90ef-f59dbef4d404" (UID: "6885e4a3-3f18-477a-90ef-f59dbef4d404"). InnerVolumeSpecName "pvc-4491ac70-4597-43a7-a792-f0813314e433". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:10:03.420070 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.420047 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6885e4a3-3f18-477a-90ef-f59dbef4d404-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:03.420070 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.420069 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/6885e4a3-3f18-477a-90ef-f59dbef4d404-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:03.420191 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.420080 5274 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/6885e4a3-3f18-477a-90ef-f59dbef4d404-gateway-rbac\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:03.420191 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.420089 5274 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/6885e4a3-3f18-477a-90ef-f59dbef4d404-gateway-tenants\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:03.420191 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.420101 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfm2t\" (UniqueName: \"kubernetes.io/projected/6885e4a3-3f18-477a-90ef-f59dbef4d404-kube-api-access-qfm2t\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:03.420191 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.420109 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/6885e4a3-3f18-477a-90ef-f59dbef4d404-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:03.420191 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.420132 5274 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" " Oct 31 00:10:03.420191 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.420142 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/6885e4a3-3f18-477a-90ef-f59dbef4d404-tempo-conf\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:03.420191 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.420151 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/6885e4a3-3f18-477a-90ef-f59dbef4d404-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:03.428261 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.428247 5274 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-4491ac70-4597-43a7-a792-f0813314e433" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062") on node "ip-10-0-129-70.us-east-2.compute.internal" Oct 31 00:10:03.486457 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.486431 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="406bd387-d4a8-4388-a6dd-cc38c86d36fd" path="/var/lib/kubelet/pods/406bd387-d4a8-4388-a6dd-cc38c86d36fd/volumes" Oct 31 00:10:03.486845 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.486832 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dd882e7-9623-45d4-96f1-0cd002fb7906" path="/var/lib/kubelet/pods/6dd882e7-9623-45d4-96f1-0cd002fb7906/volumes" Oct 31 00:10:03.487573 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.487559 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f0a400e-5e95-4403-af53-6108a54263ca" path="/var/lib/kubelet/pods/8f0a400e-5e95-4403-af53-6108a54263ca/volumes" Oct 31 00:10:03.487906 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.487895 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c14cc7ba-1add-4829-806a-639de0bc93eb" path="/var/lib/kubelet/pods/c14cc7ba-1add-4829-806a-639de0bc93eb/volumes" Oct 31 00:10:03.520821 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.520796 5274 reconciler_common.go:299] "Volume detached for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"csi-9ce3c80b5d5deac9dfdb953db4fb017080b654a85db76a2937e67cd541d89057\"" Oct 31 00:10:03.554563 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.554541 5274 generic.go:356] "Generic (PLEG): container finished" podID="6885e4a3-3f18-477a-90ef-f59dbef4d404" containerID="0999e1bfd89f852a24dd8bc572c374cce7bb0b18b8f7d475cd96a3cc548e1229" exitCode=0 Oct 31 00:10:03.554641 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.554616 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:10:03.554641 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.554617 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"6885e4a3-3f18-477a-90ef-f59dbef4d404","Type":"ContainerDied","Data":"0999e1bfd89f852a24dd8bc572c374cce7bb0b18b8f7d475cd96a3cc548e1229"} Oct 31 00:10:03.554737 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.554660 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"6885e4a3-3f18-477a-90ef-f59dbef4d404","Type":"ContainerDied","Data":"552c524b296d04c810417b3b056cb2eac35303c63b2b364b05eb6660b177a7c9"} Oct 31 00:10:03.554737 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.554682 5274 scope.go:122] "RemoveContainer" containerID="db4bad30802608be7167243b935f595c692c98e9a2700be82e79a0e2173581af" Oct 31 00:10:03.563244 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.563230 5274 scope.go:122] "RemoveContainer" containerID="0999e1bfd89f852a24dd8bc572c374cce7bb0b18b8f7d475cd96a3cc548e1229" Oct 31 00:10:03.571091 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.571077 5274 scope.go:122] "RemoveContainer" containerID="eff1b373ae1994af65d6c6cdde9b0a7c7fd4a2524c98d0c10dd6e9f198178298" Oct 31 00:10:03.574329 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.574283 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:10:03.579606 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.579579 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:10:03.580012 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.579994 5274 scope.go:122] "RemoveContainer" containerID="db4bad30802608be7167243b935f595c692c98e9a2700be82e79a0e2173581af" Oct 31 00:10:03.580272 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:10:03.580253 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db4bad30802608be7167243b935f595c692c98e9a2700be82e79a0e2173581af\": container with ID starting with db4bad30802608be7167243b935f595c692c98e9a2700be82e79a0e2173581af not found: ID does not exist" containerID="db4bad30802608be7167243b935f595c692c98e9a2700be82e79a0e2173581af" Oct 31 00:10:03.580345 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.580283 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db4bad30802608be7167243b935f595c692c98e9a2700be82e79a0e2173581af"} err="failed to get container status \"db4bad30802608be7167243b935f595c692c98e9a2700be82e79a0e2173581af\": rpc error: code = NotFound desc = could not find container \"db4bad30802608be7167243b935f595c692c98e9a2700be82e79a0e2173581af\": container with ID starting with db4bad30802608be7167243b935f595c692c98e9a2700be82e79a0e2173581af not found: ID does not exist" Oct 31 00:10:03.580345 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.580307 5274 scope.go:122] "RemoveContainer" containerID="0999e1bfd89f852a24dd8bc572c374cce7bb0b18b8f7d475cd96a3cc548e1229" Oct 31 00:10:03.580555 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:10:03.580536 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0999e1bfd89f852a24dd8bc572c374cce7bb0b18b8f7d475cd96a3cc548e1229\": container with ID starting with 0999e1bfd89f852a24dd8bc572c374cce7bb0b18b8f7d475cd96a3cc548e1229 not found: ID does not exist" containerID="0999e1bfd89f852a24dd8bc572c374cce7bb0b18b8f7d475cd96a3cc548e1229" Oct 31 00:10:03.580597 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.580562 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0999e1bfd89f852a24dd8bc572c374cce7bb0b18b8f7d475cd96a3cc548e1229"} err="failed to get container status \"0999e1bfd89f852a24dd8bc572c374cce7bb0b18b8f7d475cd96a3cc548e1229\": rpc error: code = NotFound desc = could not find container \"0999e1bfd89f852a24dd8bc572c374cce7bb0b18b8f7d475cd96a3cc548e1229\": container with ID starting with 0999e1bfd89f852a24dd8bc572c374cce7bb0b18b8f7d475cd96a3cc548e1229 not found: ID does not exist" Oct 31 00:10:03.580597 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.580577 5274 scope.go:122] "RemoveContainer" containerID="eff1b373ae1994af65d6c6cdde9b0a7c7fd4a2524c98d0c10dd6e9f198178298" Oct 31 00:10:03.580858 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:10:03.580842 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eff1b373ae1994af65d6c6cdde9b0a7c7fd4a2524c98d0c10dd6e9f198178298\": container with ID starting with eff1b373ae1994af65d6c6cdde9b0a7c7fd4a2524c98d0c10dd6e9f198178298 not found: ID does not exist" containerID="eff1b373ae1994af65d6c6cdde9b0a7c7fd4a2524c98d0c10dd6e9f198178298" Oct 31 00:10:03.580912 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:03.580863 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eff1b373ae1994af65d6c6cdde9b0a7c7fd4a2524c98d0c10dd6e9f198178298"} err="failed to get container status \"eff1b373ae1994af65d6c6cdde9b0a7c7fd4a2524c98d0c10dd6e9f198178298\": rpc error: code = NotFound desc = could not find container \"eff1b373ae1994af65d6c6cdde9b0a7c7fd4a2524c98d0c10dd6e9f198178298\": container with ID starting with eff1b373ae1994af65d6c6cdde9b0a7c7fd4a2524c98d0c10dd6e9f198178298 not found: ID does not exist" Oct 31 00:10:05.481421 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:05.481385 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6885e4a3-3f18-477a-90ef-f59dbef4d404" path="/var/lib/kubelet/pods/6885e4a3-3f18-477a-90ef-f59dbef4d404/volumes" Oct 31 00:10:09.537988 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.537877 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-lbhb9"] Oct 31 00:10:09.538404 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.538382 5274 cpu_manager.go:414] "containerMap: removing container" podUID="6dd882e7-9623-45d4-96f1-0cd002fb7906" containerName="thanos-sidecar" Oct 31 00:10:09.538444 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.538408 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dd882e7-9623-45d4-96f1-0cd002fb7906" containerName="thanos-sidecar" Oct 31 00:10:09.538444 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.538440 5274 cpu_manager.go:414] "containerMap: removing container" podUID="c14cc7ba-1add-4829-806a-639de0bc93eb" containerName="config-reloader" Oct 31 00:10:09.538520 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.538449 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c14cc7ba-1add-4829-806a-639de0bc93eb" containerName="config-reloader" Oct 31 00:10:09.538520 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.538461 5274 cpu_manager.go:414] "containerMap: removing container" podUID="c14cc7ba-1add-4829-806a-639de0bc93eb" containerName="alertmanager" Oct 31 00:10:09.538520 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.538469 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c14cc7ba-1add-4829-806a-639de0bc93eb" containerName="alertmanager" Oct 31 00:10:09.538520 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.538478 5274 cpu_manager.go:414] "containerMap: removing container" podUID="6dd882e7-9623-45d4-96f1-0cd002fb7906" containerName="prometheus" Oct 31 00:10:09.538520 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.538486 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dd882e7-9623-45d4-96f1-0cd002fb7906" containerName="prometheus" Oct 31 00:10:09.538520 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.538506 5274 cpu_manager.go:414] "containerMap: removing container" podUID="6dd882e7-9623-45d4-96f1-0cd002fb7906" containerName="init-config-reloader" Oct 31 00:10:09.538520 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.538513 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dd882e7-9623-45d4-96f1-0cd002fb7906" containerName="init-config-reloader" Oct 31 00:10:09.538741 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.538538 5274 cpu_manager.go:414] "containerMap: removing container" podUID="406bd387-d4a8-4388-a6dd-cc38c86d36fd" containerName="korrel8r" Oct 31 00:10:09.538741 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.538546 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="406bd387-d4a8-4388-a6dd-cc38c86d36fd" containerName="korrel8r" Oct 31 00:10:09.538741 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.538558 5274 cpu_manager.go:414] "containerMap: removing container" podUID="8f0a400e-5e95-4403-af53-6108a54263ca" containerName="kube-rbac-proxy" Oct 31 00:10:09.538741 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.538565 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f0a400e-5e95-4403-af53-6108a54263ca" containerName="kube-rbac-proxy" Oct 31 00:10:09.538741 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.538576 5274 cpu_manager.go:414] "containerMap: removing container" podUID="6885e4a3-3f18-477a-90ef-f59dbef4d404" containerName="tempo-gateway-opa" Oct 31 00:10:09.538741 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.538583 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6885e4a3-3f18-477a-90ef-f59dbef4d404" containerName="tempo-gateway-opa" Oct 31 00:10:09.538741 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.538599 5274 cpu_manager.go:414] "containerMap: removing container" podUID="c14cc7ba-1add-4829-806a-639de0bc93eb" containerName="init-config-reloader" Oct 31 00:10:09.538741 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.538606 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c14cc7ba-1add-4829-806a-639de0bc93eb" containerName="init-config-reloader" Oct 31 00:10:09.538741 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.538617 5274 cpu_manager.go:414] "containerMap: removing container" podUID="6885e4a3-3f18-477a-90ef-f59dbef4d404" containerName="tempo" Oct 31 00:10:09.538741 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.538624 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6885e4a3-3f18-477a-90ef-f59dbef4d404" containerName="tempo" Oct 31 00:10:09.538741 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.538636 5274 cpu_manager.go:414] "containerMap: removing container" podUID="6885e4a3-3f18-477a-90ef-f59dbef4d404" containerName="tempo-gateway" Oct 31 00:10:09.538741 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.538643 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6885e4a3-3f18-477a-90ef-f59dbef4d404" containerName="tempo-gateway" Oct 31 00:10:09.538741 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.538652 5274 cpu_manager.go:414] "containerMap: removing container" podUID="6dd882e7-9623-45d4-96f1-0cd002fb7906" containerName="config-reloader" Oct 31 00:10:09.538741 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.538659 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dd882e7-9623-45d4-96f1-0cd002fb7906" containerName="config-reloader" Oct 31 00:10:09.539186 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.538745 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dd882e7-9623-45d4-96f1-0cd002fb7906" containerName="config-reloader" Oct 31 00:10:09.539186 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.538758 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="6885e4a3-3f18-477a-90ef-f59dbef4d404" containerName="tempo" Oct 31 00:10:09.539186 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.538769 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="6885e4a3-3f18-477a-90ef-f59dbef4d404" containerName="tempo-gateway" Oct 31 00:10:09.539186 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.538778 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dd882e7-9623-45d4-96f1-0cd002fb7906" containerName="thanos-sidecar" Oct 31 00:10:09.539186 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.538789 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="c14cc7ba-1add-4829-806a-639de0bc93eb" containerName="config-reloader" Oct 31 00:10:09.539186 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.538799 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="c14cc7ba-1add-4829-806a-639de0bc93eb" containerName="alertmanager" Oct 31 00:10:09.539186 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.538807 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dd882e7-9623-45d4-96f1-0cd002fb7906" containerName="prometheus" Oct 31 00:10:09.539186 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.538820 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="406bd387-d4a8-4388-a6dd-cc38c86d36fd" containerName="korrel8r" Oct 31 00:10:09.539186 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.538827 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f0a400e-5e95-4403-af53-6108a54263ca" containerName="kube-rbac-proxy" Oct 31 00:10:09.539186 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.538834 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="6885e4a3-3f18-477a-90ef-f59dbef4d404" containerName="tempo-gateway-opa" Oct 31 00:10:09.542064 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.542045 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-lbhb9" Oct 31 00:10:09.549462 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.549440 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 31 00:10:09.549584 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.549544 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\"" Oct 31 00:10:09.550944 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.550646 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\"" Oct 31 00:10:09.552790 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.552769 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-lbhb9"] Oct 31 00:10:09.552921 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.552854 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-82qhz\"" Oct 31 00:10:09.553463 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.553439 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:10:09.553551 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.553539 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:10:09.553597 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.553588 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:10:09.612682 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.612653 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-xprkc"] Oct 31 00:10:09.616004 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.615981 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xprkc" Oct 31 00:10:09.626570 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.626548 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-xprkc"] Oct 31 00:10:09.649923 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.649898 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 31 00:10:09.650126 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.650103 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 31 00:10:09.650204 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.650190 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-42znn\"" Oct 31 00:10:09.664463 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.664442 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-lbhb9\" (UID: \"b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-lbhb9" Oct 31 00:10:09.664562 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.664490 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-lbhb9\" (UID: \"b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-lbhb9" Oct 31 00:10:09.664562 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.664550 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg4rc\" (UniqueName: \"kubernetes.io/projected/b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b-kube-api-access-pg4rc\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-lbhb9\" (UID: \"b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-lbhb9" Oct 31 00:10:09.664638 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.664585 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-lbhb9\" (UID: \"b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-lbhb9" Oct 31 00:10:09.664671 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.664652 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-lbhb9\" (UID: \"b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-lbhb9" Oct 31 00:10:09.765087 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.765054 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-lbhb9\" (UID: \"b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-lbhb9" Oct 31 00:10:09.765249 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.765103 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e7d2bd59-8c16-42a5-b59a-5f16cc966cfa-tls\") pod \"korrel8r-5cbcc89b8f-xprkc\" (UID: \"e7d2bd59-8c16-42a5-b59a-5f16cc966cfa\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xprkc" Oct 31 00:10:09.765348 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.765321 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-lbhb9\" (UID: \"b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-lbhb9" Oct 31 00:10:09.765561 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.765531 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-lbhb9\" (UID: \"b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-lbhb9" Oct 31 00:10:09.765668 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.765576 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm449\" (UniqueName: \"kubernetes.io/projected/e7d2bd59-8c16-42a5-b59a-5f16cc966cfa-kube-api-access-jm449\") pod \"korrel8r-5cbcc89b8f-xprkc\" (UID: \"e7d2bd59-8c16-42a5-b59a-5f16cc966cfa\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xprkc" Oct 31 00:10:09.765668 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.765631 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e7d2bd59-8c16-42a5-b59a-5f16cc966cfa-config\") pod \"korrel8r-5cbcc89b8f-xprkc\" (UID: \"e7d2bd59-8c16-42a5-b59a-5f16cc966cfa\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xprkc" Oct 31 00:10:09.765668 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.765657 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-lbhb9\" (UID: \"b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-lbhb9" Oct 31 00:10:09.765838 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.765723 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg4rc\" (UniqueName: \"kubernetes.io/projected/b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b-kube-api-access-pg4rc\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-lbhb9\" (UID: \"b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-lbhb9" Oct 31 00:10:09.766063 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.766040 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-lbhb9\" (UID: \"b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-lbhb9" Oct 31 00:10:09.767677 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.767649 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-lbhb9\" (UID: \"b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-lbhb9" Oct 31 00:10:09.768015 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.767994 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-lbhb9\" (UID: \"b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-lbhb9" Oct 31 00:10:09.768064 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.768011 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-lbhb9\" (UID: \"b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-lbhb9" Oct 31 00:10:09.778462 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.778442 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg4rc\" (UniqueName: \"kubernetes.io/projected/b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b-kube-api-access-pg4rc\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-lbhb9\" (UID: \"b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-lbhb9" Oct 31 00:10:09.856994 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.856902 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-lbhb9" Oct 31 00:10:09.866932 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.866910 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e7d2bd59-8c16-42a5-b59a-5f16cc966cfa-tls\") pod \"korrel8r-5cbcc89b8f-xprkc\" (UID: \"e7d2bd59-8c16-42a5-b59a-5f16cc966cfa\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xprkc" Oct 31 00:10:09.867076 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.866986 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm449\" (UniqueName: \"kubernetes.io/projected/e7d2bd59-8c16-42a5-b59a-5f16cc966cfa-kube-api-access-jm449\") pod \"korrel8r-5cbcc89b8f-xprkc\" (UID: \"e7d2bd59-8c16-42a5-b59a-5f16cc966cfa\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xprkc" Oct 31 00:10:09.867076 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.867033 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e7d2bd59-8c16-42a5-b59a-5f16cc966cfa-config\") pod \"korrel8r-5cbcc89b8f-xprkc\" (UID: \"e7d2bd59-8c16-42a5-b59a-5f16cc966cfa\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xprkc" Oct 31 00:10:09.867824 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.867801 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e7d2bd59-8c16-42a5-b59a-5f16cc966cfa-config\") pod \"korrel8r-5cbcc89b8f-xprkc\" (UID: \"e7d2bd59-8c16-42a5-b59a-5f16cc966cfa\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xprkc" Oct 31 00:10:09.869496 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.869479 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e7d2bd59-8c16-42a5-b59a-5f16cc966cfa-tls\") pod \"korrel8r-5cbcc89b8f-xprkc\" (UID: \"e7d2bd59-8c16-42a5-b59a-5f16cc966cfa\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xprkc" Oct 31 00:10:09.879600 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.879577 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm449\" (UniqueName: \"kubernetes.io/projected/e7d2bd59-8c16-42a5-b59a-5f16cc966cfa-kube-api-access-jm449\") pod \"korrel8r-5cbcc89b8f-xprkc\" (UID: \"e7d2bd59-8c16-42a5-b59a-5f16cc966cfa\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xprkc" Oct 31 00:10:09.929443 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:09.929413 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xprkc" Oct 31 00:10:10.143949 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:10.143918 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-lbhb9"] Oct 31 00:10:10.145138 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:10:10.145104 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b97c7a_e2ac_4e03_8374_92ee0ec0ff0b.slice/crio-ed84bae0cbe0b63afef44ea48c407cc14a7ab99a62d28f729e37476dd156eb5d WatchSource:0}: Error finding container ed84bae0cbe0b63afef44ea48c407cc14a7ab99a62d28f729e37476dd156eb5d: Status 404 returned error can't find the container with id ed84bae0cbe0b63afef44ea48c407cc14a7ab99a62d28f729e37476dd156eb5d Oct 31 00:10:10.207916 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:10.207885 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-xprkc"] Oct 31 00:10:10.210185 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:10:10.210152 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7d2bd59_8c16_42a5_b59a_5f16cc966cfa.slice/crio-5b7519b91e1435c469dff8601198b2c08bdddb3c62aeadba6f7d05c2659e7857 WatchSource:0}: Error finding container 5b7519b91e1435c469dff8601198b2c08bdddb3c62aeadba6f7d05c2659e7857: Status 404 returned error can't find the container with id 5b7519b91e1435c469dff8601198b2c08bdddb3c62aeadba6f7d05c2659e7857 Oct 31 00:10:10.599150 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:10.599112 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xprkc" event={"ID":"e7d2bd59-8c16-42a5-b59a-5f16cc966cfa","Type":"ContainerStarted","Data":"2ba14b6f3b43851ff8113b9cff632a0ce2eb3ff9391fa127229ab016ff10faf3"} Oct 31 00:10:10.599553 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:10.599176 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xprkc" event={"ID":"e7d2bd59-8c16-42a5-b59a-5f16cc966cfa","Type":"ContainerStarted","Data":"5b7519b91e1435c469dff8601198b2c08bdddb3c62aeadba6f7d05c2659e7857"} Oct 31 00:10:10.599553 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:10.599213 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xprkc" Oct 31 00:10:10.600263 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:10.600241 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-lbhb9" event={"ID":"b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b","Type":"ContainerStarted","Data":"ed84bae0cbe0b63afef44ea48c407cc14a7ab99a62d28f729e37476dd156eb5d"} Oct 31 00:10:10.616944 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:10.616903 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xprkc" podStartSLOduration=1.61688861 podStartE2EDuration="1.61688861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:10.616678133 +0000 UTC m=+647.253496844" watchObservedRunningTime="2026-10-31 00:10:10.61688861 +0000 UTC m=+647.253707327" Oct 31 00:10:11.607167 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:11.607135 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-lbhb9" event={"ID":"b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b","Type":"ContainerStarted","Data":"849d4610f9856044f068bd85de6a43ba6737082ac4890701321add9a6c348c78"} Oct 31 00:10:11.607612 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:11.607315 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-lbhb9" Oct 31 00:10:11.627800 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:11.627758 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-lbhb9" podStartSLOduration=2.627746544 podStartE2EDuration="2.627746544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:11.626059106 +0000 UTC m=+648.262877824" watchObservedRunningTime="2026-10-31 00:10:11.627746544 +0000 UTC m=+648.264565253" Oct 31 00:10:16.123161 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.123129 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:10:16.127270 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.127250 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:10:16.127393 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.127373 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:16.131603 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.131586 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:10:16.134041 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.134020 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:10:16.134136 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.134018 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 31 00:10:16.136547 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.136529 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:10:16.136757 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.136738 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-n9gt9\"" Oct 31 00:10:16.137855 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.137834 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 31 00:10:16.137977 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.137859 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 31 00:10:16.138027 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.137936 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 31 00:10:16.144652 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.144632 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:10:16.146186 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.146165 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:10:16.234691 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.234660 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 31 00:10:16.234691 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.234661 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 31 00:10:16.234691 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.234669 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 31 00:10:16.234916 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.234671 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 31 00:10:16.234916 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.234732 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 31 00:10:16.234916 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.234776 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:10:16.234916 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.234779 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-qnf9d\"" Oct 31 00:10:16.316359 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.316325 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:16.316359 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.316361 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:10:16.316527 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.316383 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g94lz\" (UniqueName: \"kubernetes.io/projected/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-kube-api-access-g94lz\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:16.316527 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.316494 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:10:16.316527 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.316518 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:16.316664 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.316542 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:16.316664 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.316598 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:16.316664 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.316632 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:16.316760 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.316738 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:10:16.316793 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.316770 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:16.316835 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.316801 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:16.316884 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.316850 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:10:16.316884 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.316868 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:16.317008 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.316885 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt6zz\" (UniqueName: \"kubernetes.io/projected/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-kube-api-access-tt6zz\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:10:16.317008 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.316907 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:10:16.317008 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.316933 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:16.317152 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.317049 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:10:16.317152 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.317076 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:16.317152 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.317094 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:16.418063 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.417988 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:16.418063 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.418024 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:10:16.418063 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.418044 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:16.418063 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.418061 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt6zz\" (UniqueName: \"kubernetes.io/projected/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-kube-api-access-tt6zz\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:10:16.418374 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.418081 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:10:16.418374 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.418101 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:16.418374 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.418146 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:10:16.418374 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.418188 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:16.418374 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.418339 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:16.418652 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.418399 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:16.418652 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.418434 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:10:16.418652 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.418458 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g94lz\" (UniqueName: \"kubernetes.io/projected/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-kube-api-access-g94lz\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:16.418652 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.418499 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:10:16.418652 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.418523 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:16.418652 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.418553 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:16.418652 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.418590 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:16.418652 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.418616 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:16.418652 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.418646 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:10:16.419140 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.418689 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:16.419140 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.418791 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:10:16.419140 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.419051 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:16.419601 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.419574 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:16.419762 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.419694 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:16.421624 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.421596 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:16.421730 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.421599 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:10:16.421730 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.421675 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:16.422288 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.421992 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:10:16.423593 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.423271 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:16.423593 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.423276 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:16.424099 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.424073 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:16.424636 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.424561 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:16.424636 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.424567 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:10:16.424636 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.424592 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:10:16.424832 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.424659 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:10:16.424832 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.424807 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:16.425731 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.425707 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:16.432550 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.432525 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt6zz\" (UniqueName: \"kubernetes.io/projected/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-kube-api-access-tt6zz\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:10:16.435022 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.435005 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g94lz\" (UniqueName: \"kubernetes.io/projected/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-kube-api-access-g94lz\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:16.460024 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.459991 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:10:16.741276 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:16.741253 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:10:16.743605 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:10:16.743575 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72cfcd9d_cf49_4b29_b378_dec7b4b3dfe6.slice/crio-70296fd8d231724b8706579b20adfa128d8990dfbef5971192987a92425006d4 WatchSource:0}: Error finding container 70296fd8d231724b8706579b20adfa128d8990dfbef5971192987a92425006d4: Status 404 returned error can't find the container with id 70296fd8d231724b8706579b20adfa128d8990dfbef5971192987a92425006d4 Oct 31 00:10:17.614085 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:17.614060 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-lbhb9" Oct 31 00:10:17.641277 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:17.641240 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6","Type":"ContainerStarted","Data":"aeb605a44bc2f84f9a3d05f7bc0a8af46fd1f72351fe75d9f2eb966cc5ca63a6"} Oct 31 00:10:17.641409 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:17.641314 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6","Type":"ContainerStarted","Data":"70296fd8d231724b8706579b20adfa128d8990dfbef5971192987a92425006d4"} Oct 31 00:10:18.537861 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:18.537825 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:18.544355 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:18.544330 5274 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:18.638792 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:18.638763 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:18.639133 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:18.638857 5274 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:18.642158 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:18.642130 5274 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") DevicePath \"csi-51e8027f163b5f98040faf3f8c30848dd44a7a90ca2bd0af2446ea6398ed018d\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:18.734379 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:18.734345 5274 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/53bcf0e8434a816dcc013cea20657add42b4da36ecdc0fff46af7413bc0037a5/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:18.749854 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:18.749829 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:18.851720 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:18.851649 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:19.136107 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:19.136077 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:10:19.137074 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:10:19.137044 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda75a1126_5bff_41dd_a732_5a24e6b4d5e5.slice/crio-125bd07a37dc61a8c256b88c6250b98396a9bbde3086f54243ba9fc4437c4b75 WatchSource:0}: Error finding container 125bd07a37dc61a8c256b88c6250b98396a9bbde3086f54243ba9fc4437c4b75: Status 404 returned error can't find the container with id 125bd07a37dc61a8c256b88c6250b98396a9bbde3086f54243ba9fc4437c4b75 Oct 31 00:10:19.653154 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:19.653120 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"a75a1126-5bff-41dd-a732-5a24e6b4d5e5","Type":"ContainerStarted","Data":"125bd07a37dc61a8c256b88c6250b98396a9bbde3086f54243ba9fc4437c4b75"} Oct 31 00:10:20.660608 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:20.660568 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"a75a1126-5bff-41dd-a732-5a24e6b4d5e5","Type":"ContainerStarted","Data":"76493e766a972779c8d756016833221c293a42d884ec55d4025e79273ee7dbde"} Oct 31 00:10:20.897795 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:10:20.897760 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:10:21.609128 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:21.609101 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xprkc" Oct 31 00:10:21.667846 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:21.667809 5274 generic.go:356] "Generic (PLEG): container finished" podID="72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6" containerID="aeb605a44bc2f84f9a3d05f7bc0a8af46fd1f72351fe75d9f2eb966cc5ca63a6" exitCode=0 Oct 31 00:10:21.668246 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:21.667879 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6","Type":"ContainerDied","Data":"aeb605a44bc2f84f9a3d05f7bc0a8af46fd1f72351fe75d9f2eb966cc5ca63a6"} Oct 31 00:10:22.675112 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:22.675080 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6","Type":"ContainerStarted","Data":"d8231d74911432ff28005561f0bc6c10dd90d81a1ea1343493ce000faaba01cd"} Oct 31 00:10:22.675559 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:22.675182 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6","Type":"ContainerStarted","Data":"91300aef897dfaf7e7d7936ea65f8b0b9f217ff5a63fba306c3e7ef5ae49e799"} Oct 31 00:10:22.675559 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:22.675397 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:10:22.703711 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:22.703662 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=2.9178462019999998 podStartE2EDuration="6.703650847s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.785804645s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:22.701716688 +0000 UTC m=+659.338535407" watchObservedRunningTime="2026-10-31 00:10:22.703650847 +0000 UTC m=+659.340469563" Oct 31 00:10:23.680572 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:23.680546 5274 generic.go:356] "Generic (PLEG): container finished" podID="a75a1126-5bff-41dd-a732-5a24e6b4d5e5" containerID="76493e766a972779c8d756016833221c293a42d884ec55d4025e79273ee7dbde" exitCode=0 Oct 31 00:10:23.680866 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:23.680614 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"a75a1126-5bff-41dd-a732-5a24e6b4d5e5","Type":"ContainerDied","Data":"76493e766a972779c8d756016833221c293a42d884ec55d4025e79273ee7dbde"} Oct 31 00:10:24.690508 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:24.690479 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"a75a1126-5bff-41dd-a732-5a24e6b4d5e5","Type":"ContainerStarted","Data":"b8a139d3b41947671950fac51a9ed64f93590f1f2934855c1b9895e63159d0b5"} Oct 31 00:10:24.690849 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:24.690548 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"a75a1126-5bff-41dd-a732-5a24e6b4d5e5","Type":"ContainerStarted","Data":"756ebf069641c84d785c4e79f4a89a1514e483ab40ff6e741113462d4895edb6"} Oct 31 00:10:24.690849 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:24.690574 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"a75a1126-5bff-41dd-a732-5a24e6b4d5e5","Type":"ContainerStarted","Data":"0c0cacbbc110c969a184fbd7d90cb2cab919b1678d367df996c58c1f93665e2f"} Oct 31 00:10:24.723565 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:24.723514 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=5.43899102 podStartE2EDuration="8.723501466s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.284510446s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:10:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:24.721812776 +0000 UTC m=+661.358631492" watchObservedRunningTime="2026-10-31 00:10:24.723501466 +0000 UTC m=+661.360320181" Oct 31 00:10:28.682175 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:28.682139 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:10:28.852429 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:28.852397 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:33.852749 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:33.852720 5274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:33.858020 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:33.857999 5274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:34.749295 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:10:34.749268 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:10:50.898575 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:10:50.898534 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:11:20.898767 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:11:20.898734 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:11:50.898532 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:11:50.898495 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:12:16.238081 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:16.238047 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-24bxz"] Oct 31 00:12:16.241392 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:16.241376 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-24bxz" Oct 31 00:12:16.250610 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:16.250582 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-24bxz"] Oct 31 00:12:16.266262 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:16.266238 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4c2b4fae-0239-488e-bfc0-b11500d74773-config\") pod \"korrel8r-66d6f7f696-24bxz\" (UID: \"4c2b4fae-0239-488e-bfc0-b11500d74773\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-24bxz" Oct 31 00:12:16.266422 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:16.266283 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/4c2b4fae-0239-488e-bfc0-b11500d74773-tls\") pod \"korrel8r-66d6f7f696-24bxz\" (UID: \"4c2b4fae-0239-488e-bfc0-b11500d74773\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-24bxz" Oct 31 00:12:16.266422 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:16.266339 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqw8l\" (UniqueName: \"kubernetes.io/projected/4c2b4fae-0239-488e-bfc0-b11500d74773-kube-api-access-hqw8l\") pod \"korrel8r-66d6f7f696-24bxz\" (UID: \"4c2b4fae-0239-488e-bfc0-b11500d74773\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-24bxz" Oct 31 00:12:16.366796 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:16.366759 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4c2b4fae-0239-488e-bfc0-b11500d74773-config\") pod \"korrel8r-66d6f7f696-24bxz\" (UID: \"4c2b4fae-0239-488e-bfc0-b11500d74773\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-24bxz" Oct 31 00:12:16.366993 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:16.366818 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/4c2b4fae-0239-488e-bfc0-b11500d74773-tls\") pod \"korrel8r-66d6f7f696-24bxz\" (UID: \"4c2b4fae-0239-488e-bfc0-b11500d74773\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-24bxz" Oct 31 00:12:16.366993 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:16.366869 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqw8l\" (UniqueName: \"kubernetes.io/projected/4c2b4fae-0239-488e-bfc0-b11500d74773-kube-api-access-hqw8l\") pod \"korrel8r-66d6f7f696-24bxz\" (UID: \"4c2b4fae-0239-488e-bfc0-b11500d74773\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-24bxz" Oct 31 00:12:16.367801 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:16.367776 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4c2b4fae-0239-488e-bfc0-b11500d74773-config\") pod \"korrel8r-66d6f7f696-24bxz\" (UID: \"4c2b4fae-0239-488e-bfc0-b11500d74773\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-24bxz" Oct 31 00:12:16.369206 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:16.369187 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/4c2b4fae-0239-488e-bfc0-b11500d74773-tls\") pod \"korrel8r-66d6f7f696-24bxz\" (UID: \"4c2b4fae-0239-488e-bfc0-b11500d74773\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-24bxz" Oct 31 00:12:16.379354 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:16.379333 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqw8l\" (UniqueName: \"kubernetes.io/projected/4c2b4fae-0239-488e-bfc0-b11500d74773-kube-api-access-hqw8l\") pod \"korrel8r-66d6f7f696-24bxz\" (UID: \"4c2b4fae-0239-488e-bfc0-b11500d74773\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-24bxz" Oct 31 00:12:16.554862 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:16.554781 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-24bxz" Oct 31 00:12:16.828717 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:16.828680 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-24bxz"] Oct 31 00:12:16.831939 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:12:16.831913 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c2b4fae_0239_488e_bfc0_b11500d74773.slice/crio-798a364bfa9ece85b2e3238dcb00a8ff8179638a0f262427ae3ecbe73cf9b01a WatchSource:0}: Error finding container 798a364bfa9ece85b2e3238dcb00a8ff8179638a0f262427ae3ecbe73cf9b01a: Status 404 returned error can't find the container with id 798a364bfa9ece85b2e3238dcb00a8ff8179638a0f262427ae3ecbe73cf9b01a Oct 31 00:12:17.237442 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:17.237405 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-24bxz" event={"ID":"4c2b4fae-0239-488e-bfc0-b11500d74773","Type":"ContainerStarted","Data":"acd428f423c2f6d58e06de71d02e95255ae880981c6b789e6016c4dc9c6fb27b"} Oct 31 00:12:17.237619 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:17.237476 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-24bxz" Oct 31 00:12:17.237619 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:17.237499 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-24bxz" event={"ID":"4c2b4fae-0239-488e-bfc0-b11500d74773","Type":"ContainerStarted","Data":"798a364bfa9ece85b2e3238dcb00a8ff8179638a0f262427ae3ecbe73cf9b01a"} Oct 31 00:12:17.288888 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:17.288844 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-24bxz" podStartSLOduration=1.288830115 podStartE2EDuration="1.288830115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:17.286378191 +0000 UTC m=+773.923196909" watchObservedRunningTime="2026-10-31 00:12:17.288830115 +0000 UTC m=+773.925648831" Oct 31 00:12:18.422122 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:18.422089 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:12:18.425835 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:18.425818 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:18.429877 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:18.429847 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\"" Oct 31 00:12:18.429877 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:18.429859 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 31 00:12:18.430539 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:18.430514 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\"" Oct 31 00:12:18.430723 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:18.430542 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\"" Oct 31 00:12:18.430723 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:18.430550 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 31 00:12:18.430723 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:18.430551 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\"" Oct 31 00:12:18.431008 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:18.430556 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\"" Oct 31 00:12:18.431008 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:18.430573 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-96kj2\"" Oct 31 00:12:18.439874 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:18.439852 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:12:18.581851 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:18.581824 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/85273358-99d0-4064-a532-12949bfaaba2-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:18.582027 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:18.581862 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/85273358-99d0-4064-a532-12949bfaaba2-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:18.582027 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:18.581881 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7f5c\" (UniqueName: \"kubernetes.io/projected/85273358-99d0-4064-a532-12949bfaaba2-kube-api-access-m7f5c\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:18.582027 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:18.582021 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/85273358-99d0-4064-a532-12949bfaaba2-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:18.582175 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:18.582064 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/85273358-99d0-4064-a532-12949bfaaba2-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:18.582175 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:18.582104 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/85273358-99d0-4064-a532-12949bfaaba2-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:18.582280 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:18.582232 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/85273358-99d0-4064-a532-12949bfaaba2-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:18.582280 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:18.582270 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85273358-99d0-4064-a532-12949bfaaba2-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:18.683013 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:18.682921 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/85273358-99d0-4064-a532-12949bfaaba2-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:18.683013 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:18.682980 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7f5c\" (UniqueName: \"kubernetes.io/projected/85273358-99d0-4064-a532-12949bfaaba2-kube-api-access-m7f5c\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:18.683190 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:18.683024 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/85273358-99d0-4064-a532-12949bfaaba2-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:18.683190 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:18.683145 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/85273358-99d0-4064-a532-12949bfaaba2-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:18.683321 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:18.683195 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/85273358-99d0-4064-a532-12949bfaaba2-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:18.683321 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:18.683228 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/85273358-99d0-4064-a532-12949bfaaba2-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:18.683321 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:18.683246 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85273358-99d0-4064-a532-12949bfaaba2-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:18.683321 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:18.683303 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/85273358-99d0-4064-a532-12949bfaaba2-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:18.683729 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:18.683702 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/85273358-99d0-4064-a532-12949bfaaba2-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:18.683900 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:18.683734 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/85273358-99d0-4064-a532-12949bfaaba2-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:18.683900 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:18.683836 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/85273358-99d0-4064-a532-12949bfaaba2-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:18.684029 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:18.684011 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85273358-99d0-4064-a532-12949bfaaba2-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:18.685735 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:18.685705 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/85273358-99d0-4064-a532-12949bfaaba2-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:18.685813 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:18.685762 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/85273358-99d0-4064-a532-12949bfaaba2-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:18.685813 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:18.685805 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/85273358-99d0-4064-a532-12949bfaaba2-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:18.695268 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:18.695241 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7f5c\" (UniqueName: \"kubernetes.io/projected/85273358-99d0-4064-a532-12949bfaaba2-kube-api-access-m7f5c\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:20.801589 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:20.801558 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:20.808095 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:20.808071 5274 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:20.899335 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:12:20.899304 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:12:20.902742 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:20.902719 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:20.902862 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:20.902847 5274 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:20.906187 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:20.906168 5274 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") DevicePath \"csi-9ce3c80b5d5deac9dfdb953db4fb017080b654a85db76a2937e67cd541d89057\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:20.994032 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:20.994005 5274 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/151619720cb39d40781bf258206fb0be746b77e5f55879e189e4cf0d6e46dfb0/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:21.019196 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:21.019168 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:21.143606 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:21.143576 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:21.423147 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:21.423122 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:12:21.424857 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:12:21.424827 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85273358_99d0_4064_a532_12949bfaaba2.slice/crio-71093101977e0ca31ca4fb7556f2c8e28c6b1510edaa8edb58240554707f23ff WatchSource:0}: Error finding container 71093101977e0ca31ca4fb7556f2c8e28c6b1510edaa8edb58240554707f23ff: Status 404 returned error can't find the container with id 71093101977e0ca31ca4fb7556f2c8e28c6b1510edaa8edb58240554707f23ff Oct 31 00:12:22.272938 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:22.272898 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"85273358-99d0-4064-a532-12949bfaaba2","Type":"ContainerStarted","Data":"10342dce14b9bf67d3af221eb2570fe11675463ea19e10c7ef72b38153092fa5"} Oct 31 00:12:22.273386 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:22.272984 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:22.273386 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:22.273010 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"85273358-99d0-4064-a532-12949bfaaba2","Type":"ContainerStarted","Data":"044d0657054494dab71a5b9edb54f4c5923ab1a156a00129ddff6ab398d47cc4"} Oct 31 00:12:22.273386 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:22.273033 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"85273358-99d0-4064-a532-12949bfaaba2","Type":"ContainerStarted","Data":"945282e431dad2b50fa915f70efe02dd3ce966db0506e04a93eb7c4cb52679e8"} Oct 31 00:12:22.273386 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:22.273058 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:22.273386 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:22.273081 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"85273358-99d0-4064-a532-12949bfaaba2","Type":"ContainerStarted","Data":"71093101977e0ca31ca4fb7556f2c8e28c6b1510edaa8edb58240554707f23ff"} Oct 31 00:12:22.273386 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:22.273105 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:22.275593 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:22.275572 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:22.281593 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:22.281571 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:22.300093 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:22.300046 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=4.300021315 podStartE2EDuration="4.300021315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:22.298217261 +0000 UTC m=+778.935035983" watchObservedRunningTime="2026-10-31 00:12:22.300021315 +0000 UTC m=+778.936840046" Oct 31 00:12:28.251969 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:28.251928 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-24bxz" Oct 31 00:12:28.298542 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:28.298503 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-xprkc"] Oct 31 00:12:28.298776 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:28.298746 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xprkc" podUID="e7d2bd59-8c16-42a5-b59a-5f16cc966cfa" containerName="korrel8r" containerID="cri-o://2ba14b6f3b43851ff8113b9cff632a0ce2eb3ff9391fa127229ab016ff10faf3" gracePeriod=30 Oct 31 00:12:28.553017 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:28.552991 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xprkc" Oct 31 00:12:28.573872 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:28.573832 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7d2bd59-8c16-42a5-b59a-5f16cc966cfa-tls\" (UniqueName: \"kubernetes.io/secret/e7d2bd59-8c16-42a5-b59a-5f16cc966cfa-tls\") pod \"e7d2bd59-8c16-42a5-b59a-5f16cc966cfa\" (UID: \"e7d2bd59-8c16-42a5-b59a-5f16cc966cfa\") " Oct 31 00:12:28.574042 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:28.573874 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7d2bd59-8c16-42a5-b59a-5f16cc966cfa-kube-api-access-jm449\" (UniqueName: \"kubernetes.io/projected/e7d2bd59-8c16-42a5-b59a-5f16cc966cfa-kube-api-access-jm449\") pod \"e7d2bd59-8c16-42a5-b59a-5f16cc966cfa\" (UID: \"e7d2bd59-8c16-42a5-b59a-5f16cc966cfa\") " Oct 31 00:12:28.574042 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:28.573910 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7d2bd59-8c16-42a5-b59a-5f16cc966cfa-config\" (UniqueName: \"kubernetes.io/configmap/e7d2bd59-8c16-42a5-b59a-5f16cc966cfa-config\") pod \"e7d2bd59-8c16-42a5-b59a-5f16cc966cfa\" (UID: \"e7d2bd59-8c16-42a5-b59a-5f16cc966cfa\") " Oct 31 00:12:28.574627 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:28.574595 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7d2bd59-8c16-42a5-b59a-5f16cc966cfa-config" pod "e7d2bd59-8c16-42a5-b59a-5f16cc966cfa" (UID: "e7d2bd59-8c16-42a5-b59a-5f16cc966cfa"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:12:28.576289 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:28.576263 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7d2bd59-8c16-42a5-b59a-5f16cc966cfa-kube-api-access-jm449" pod "e7d2bd59-8c16-42a5-b59a-5f16cc966cfa" (UID: "e7d2bd59-8c16-42a5-b59a-5f16cc966cfa"). InnerVolumeSpecName "kube-api-access-jm449". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:12:28.576516 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:28.576500 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7d2bd59-8c16-42a5-b59a-5f16cc966cfa-tls" pod "e7d2bd59-8c16-42a5-b59a-5f16cc966cfa" (UID: "e7d2bd59-8c16-42a5-b59a-5f16cc966cfa"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:28.675400 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:28.675374 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e7d2bd59-8c16-42a5-b59a-5f16cc966cfa-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:28.675400 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:28.675400 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm449\" (UniqueName: \"kubernetes.io/projected/e7d2bd59-8c16-42a5-b59a-5f16cc966cfa-kube-api-access-jm449\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:28.675549 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:28.675411 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e7d2bd59-8c16-42a5-b59a-5f16cc966cfa-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:28.877276 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:28.877244 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-fjl92"] Oct 31 00:12:28.877630 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:28.877618 5274 cpu_manager.go:414] "containerMap: removing container" podUID="e7d2bd59-8c16-42a5-b59a-5f16cc966cfa" containerName="korrel8r" Oct 31 00:12:28.877685 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:28.877632 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7d2bd59-8c16-42a5-b59a-5f16cc966cfa" containerName="korrel8r" Oct 31 00:12:28.877722 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:28.877695 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7d2bd59-8c16-42a5-b59a-5f16cc966cfa" containerName="korrel8r" Oct 31 00:12:28.880826 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:28.880810 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-fjl92" Oct 31 00:12:28.891347 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:28.891323 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-fjl92"] Oct 31 00:12:28.977317 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:28.977288 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/122ca78b-1834-4ae8-abb1-7962c7f5797d-tls\") pod \"korrel8r-5cbcc89b8f-fjl92\" (UID: \"122ca78b-1834-4ae8-abb1-7962c7f5797d\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-fjl92" Oct 31 00:12:28.977317 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:28.977321 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/122ca78b-1834-4ae8-abb1-7962c7f5797d-config\") pod \"korrel8r-5cbcc89b8f-fjl92\" (UID: \"122ca78b-1834-4ae8-abb1-7962c7f5797d\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-fjl92" Oct 31 00:12:28.977487 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:28.977355 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwldz\" (UniqueName: \"kubernetes.io/projected/122ca78b-1834-4ae8-abb1-7962c7f5797d-kube-api-access-dwldz\") pod \"korrel8r-5cbcc89b8f-fjl92\" (UID: \"122ca78b-1834-4ae8-abb1-7962c7f5797d\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-fjl92" Oct 31 00:12:29.078474 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:29.078439 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/122ca78b-1834-4ae8-abb1-7962c7f5797d-tls\") pod \"korrel8r-5cbcc89b8f-fjl92\" (UID: \"122ca78b-1834-4ae8-abb1-7962c7f5797d\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-fjl92" Oct 31 00:12:29.078638 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:29.078480 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/122ca78b-1834-4ae8-abb1-7962c7f5797d-config\") pod \"korrel8r-5cbcc89b8f-fjl92\" (UID: \"122ca78b-1834-4ae8-abb1-7962c7f5797d\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-fjl92" Oct 31 00:12:29.078638 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:29.078534 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwldz\" (UniqueName: \"kubernetes.io/projected/122ca78b-1834-4ae8-abb1-7962c7f5797d-kube-api-access-dwldz\") pod \"korrel8r-5cbcc89b8f-fjl92\" (UID: \"122ca78b-1834-4ae8-abb1-7962c7f5797d\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-fjl92" Oct 31 00:12:29.079380 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:29.079352 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/122ca78b-1834-4ae8-abb1-7962c7f5797d-config\") pod \"korrel8r-5cbcc89b8f-fjl92\" (UID: \"122ca78b-1834-4ae8-abb1-7962c7f5797d\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-fjl92" Oct 31 00:12:29.080901 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:29.080879 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/122ca78b-1834-4ae8-abb1-7962c7f5797d-tls\") pod \"korrel8r-5cbcc89b8f-fjl92\" (UID: \"122ca78b-1834-4ae8-abb1-7962c7f5797d\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-fjl92" Oct 31 00:12:29.090870 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:29.090848 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwldz\" (UniqueName: \"kubernetes.io/projected/122ca78b-1834-4ae8-abb1-7962c7f5797d-kube-api-access-dwldz\") pod \"korrel8r-5cbcc89b8f-fjl92\" (UID: \"122ca78b-1834-4ae8-abb1-7962c7f5797d\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-fjl92" Oct 31 00:12:29.194661 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:29.194606 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-fjl92" Oct 31 00:12:29.318941 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:29.318911 5274 generic.go:356] "Generic (PLEG): container finished" podID="e7d2bd59-8c16-42a5-b59a-5f16cc966cfa" containerID="2ba14b6f3b43851ff8113b9cff632a0ce2eb3ff9391fa127229ab016ff10faf3" exitCode=0 Oct 31 00:12:29.319333 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:29.319002 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xprkc" Oct 31 00:12:29.319333 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:29.319028 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xprkc" event={"ID":"e7d2bd59-8c16-42a5-b59a-5f16cc966cfa","Type":"ContainerDied","Data":"2ba14b6f3b43851ff8113b9cff632a0ce2eb3ff9391fa127229ab016ff10faf3"} Oct 31 00:12:29.319333 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:29.319081 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-xprkc" event={"ID":"e7d2bd59-8c16-42a5-b59a-5f16cc966cfa","Type":"ContainerDied","Data":"5b7519b91e1435c469dff8601198b2c08bdddb3c62aeadba6f7d05c2659e7857"} Oct 31 00:12:29.319333 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:29.319105 5274 scope.go:122] "RemoveContainer" containerID="2ba14b6f3b43851ff8113b9cff632a0ce2eb3ff9391fa127229ab016ff10faf3" Oct 31 00:12:29.328486 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:29.328468 5274 scope.go:122] "RemoveContainer" containerID="2ba14b6f3b43851ff8113b9cff632a0ce2eb3ff9391fa127229ab016ff10faf3" Oct 31 00:12:29.328728 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:12:29.328712 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ba14b6f3b43851ff8113b9cff632a0ce2eb3ff9391fa127229ab016ff10faf3\": container with ID starting with 2ba14b6f3b43851ff8113b9cff632a0ce2eb3ff9391fa127229ab016ff10faf3 not found: ID does not exist" containerID="2ba14b6f3b43851ff8113b9cff632a0ce2eb3ff9391fa127229ab016ff10faf3" Oct 31 00:12:29.328777 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:29.328739 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ba14b6f3b43851ff8113b9cff632a0ce2eb3ff9391fa127229ab016ff10faf3"} err="failed to get container status \"2ba14b6f3b43851ff8113b9cff632a0ce2eb3ff9391fa127229ab016ff10faf3\": rpc error: code = NotFound desc = could not find container \"2ba14b6f3b43851ff8113b9cff632a0ce2eb3ff9391fa127229ab016ff10faf3\": container with ID starting with 2ba14b6f3b43851ff8113b9cff632a0ce2eb3ff9391fa127229ab016ff10faf3 not found: ID does not exist" Oct 31 00:12:29.343273 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:29.343242 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-xprkc"] Oct 31 00:12:29.349261 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:29.349241 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-xprkc"] Oct 31 00:12:29.467798 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:29.467773 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-fjl92"] Oct 31 00:12:29.469467 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:12:29.469441 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod122ca78b_1834_4ae8_abb1_7962c7f5797d.slice/crio-8b8f4095329d3b4052ecf837975145852a7751d438839a013834509a9b69f399 WatchSource:0}: Error finding container 8b8f4095329d3b4052ecf837975145852a7751d438839a013834509a9b69f399: Status 404 returned error can't find the container with id 8b8f4095329d3b4052ecf837975145852a7751d438839a013834509a9b69f399 Oct 31 00:12:29.481947 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:29.481914 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7d2bd59-8c16-42a5-b59a-5f16cc966cfa" path="/var/lib/kubelet/pods/e7d2bd59-8c16-42a5-b59a-5f16cc966cfa/volumes" Oct 31 00:12:29.575570 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:29.575539 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:12:29.575794 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:29.575761 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="85273358-99d0-4064-a532-12949bfaaba2" containerName="tempo" containerID="cri-o://945282e431dad2b50fa915f70efe02dd3ce966db0506e04a93eb7c4cb52679e8" gracePeriod=30 Oct 31 00:12:29.575928 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:29.575798 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="85273358-99d0-4064-a532-12949bfaaba2" containerName="tempo-gateway-opa" containerID="cri-o://10342dce14b9bf67d3af221eb2570fe11675463ea19e10c7ef72b38153092fa5" gracePeriod=30 Oct 31 00:12:29.575928 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:29.575811 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="85273358-99d0-4064-a532-12949bfaaba2" containerName="tempo-gateway" containerID="cri-o://044d0657054494dab71a5b9edb54f4c5923ab1a156a00129ddff6ab398d47cc4" gracePeriod=30 Oct 31 00:12:30.326793 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:30.326755 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-fjl92" event={"ID":"122ca78b-1834-4ae8-abb1-7962c7f5797d","Type":"ContainerStarted","Data":"4eeed830b5fa1173184e535bf51d897be0033f494d163f82d4753798164a7cea"} Oct 31 00:12:30.327263 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:30.326810 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-fjl92" event={"ID":"122ca78b-1834-4ae8-abb1-7962c7f5797d","Type":"ContainerStarted","Data":"8b8f4095329d3b4052ecf837975145852a7751d438839a013834509a9b69f399"} Oct 31 00:12:30.327263 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:30.326867 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-fjl92" Oct 31 00:12:30.328935 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:30.328912 5274 generic.go:356] "Generic (PLEG): container finished" podID="85273358-99d0-4064-a532-12949bfaaba2" containerID="10342dce14b9bf67d3af221eb2570fe11675463ea19e10c7ef72b38153092fa5" exitCode=0 Oct 31 00:12:30.328935 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:30.328933 5274 generic.go:356] "Generic (PLEG): container finished" podID="85273358-99d0-4064-a532-12949bfaaba2" containerID="945282e431dad2b50fa915f70efe02dd3ce966db0506e04a93eb7c4cb52679e8" exitCode=0 Oct 31 00:12:30.329097 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:30.328983 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"85273358-99d0-4064-a532-12949bfaaba2","Type":"ContainerDied","Data":"10342dce14b9bf67d3af221eb2570fe11675463ea19e10c7ef72b38153092fa5"} Oct 31 00:12:30.329097 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:30.329023 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"85273358-99d0-4064-a532-12949bfaaba2","Type":"ContainerDied","Data":"945282e431dad2b50fa915f70efe02dd3ce966db0506e04a93eb7c4cb52679e8"} Oct 31 00:12:30.347015 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:30.346972 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-fjl92" podStartSLOduration=2.346945898 podStartE2EDuration="2.346945898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:30.34506873 +0000 UTC m=+786.981887464" watchObservedRunningTime="2026-10-31 00:12:30.346945898 +0000 UTC m=+786.983764613" Oct 31 00:12:30.835439 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:30.835414 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:30.897230 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:30.897196 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85273358-99d0-4064-a532-12949bfaaba2-kube-api-access-m7f5c\" (UniqueName: \"kubernetes.io/projected/85273358-99d0-4064-a532-12949bfaaba2-kube-api-access-m7f5c\") pod \"85273358-99d0-4064-a532-12949bfaaba2\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") " Oct 31 00:12:30.897385 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:30.897273 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85273358-99d0-4064-a532-12949bfaaba2-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/85273358-99d0-4064-a532-12949bfaaba2-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"85273358-99d0-4064-a532-12949bfaaba2\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") " Oct 31 00:12:30.897385 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:30.897302 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85273358-99d0-4064-a532-12949bfaaba2-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/85273358-99d0-4064-a532-12949bfaaba2-gateway-tenants\") pod \"85273358-99d0-4064-a532-12949bfaaba2\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") " Oct 31 00:12:30.897385 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:30.897341 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85273358-99d0-4064-a532-12949bfaaba2-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/85273358-99d0-4064-a532-12949bfaaba2-gateway-rbac\") pod \"85273358-99d0-4064-a532-12949bfaaba2\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") " Oct 31 00:12:30.897385 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:30.897372 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85273358-99d0-4064-a532-12949bfaaba2-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/85273358-99d0-4064-a532-12949bfaaba2-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"85273358-99d0-4064-a532-12949bfaaba2\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") " Oct 31 00:12:30.897653 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:30.897408 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85273358-99d0-4064-a532-12949bfaaba2-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/85273358-99d0-4064-a532-12949bfaaba2-tempo-data-science-tempomonolithic-mtls\") pod \"85273358-99d0-4064-a532-12949bfaaba2\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") " Oct 31 00:12:30.897653 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:30.897537 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"85273358-99d0-4064-a532-12949bfaaba2\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") " Oct 31 00:12:30.897653 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:30.897571 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85273358-99d0-4064-a532-12949bfaaba2-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/85273358-99d0-4064-a532-12949bfaaba2-tempo-conf\") pod \"85273358-99d0-4064-a532-12949bfaaba2\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") " Oct 31 00:12:30.897653 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:30.897600 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85273358-99d0-4064-a532-12949bfaaba2-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85273358-99d0-4064-a532-12949bfaaba2-tempo-data-science-tempomonolithic-ca-bundle\") pod \"85273358-99d0-4064-a532-12949bfaaba2\" (UID: \"85273358-99d0-4064-a532-12949bfaaba2\") " Oct 31 00:12:30.897865 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:30.897774 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85273358-99d0-4064-a532-12949bfaaba2-tempo-data-science-tempomonolithic-serving-cabundle" pod "85273358-99d0-4064-a532-12949bfaaba2" (UID: "85273358-99d0-4064-a532-12949bfaaba2"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:12:30.898086 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:30.898060 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/85273358-99d0-4064-a532-12949bfaaba2-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:30.898086 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:30.898066 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85273358-99d0-4064-a532-12949bfaaba2-gateway-rbac" pod "85273358-99d0-4064-a532-12949bfaaba2" (UID: "85273358-99d0-4064-a532-12949bfaaba2"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:12:30.898228 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:30.898109 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85273358-99d0-4064-a532-12949bfaaba2-tempo-data-science-tempomonolithic-ca-bundle" pod "85273358-99d0-4064-a532-12949bfaaba2" (UID: "85273358-99d0-4064-a532-12949bfaaba2"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:12:30.898382 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:30.898361 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85273358-99d0-4064-a532-12949bfaaba2-tempo-conf" pod "85273358-99d0-4064-a532-12949bfaaba2" (UID: "85273358-99d0-4064-a532-12949bfaaba2"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:12:30.899842 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:30.899783 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85273358-99d0-4064-a532-12949bfaaba2-kube-api-access-m7f5c" pod "85273358-99d0-4064-a532-12949bfaaba2" (UID: "85273358-99d0-4064-a532-12949bfaaba2"). InnerVolumeSpecName "kube-api-access-m7f5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:12:30.900013 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:30.899988 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85273358-99d0-4064-a532-12949bfaaba2-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "85273358-99d0-4064-a532-12949bfaaba2" (UID: "85273358-99d0-4064-a532-12949bfaaba2"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:30.900218 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:30.900191 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85273358-99d0-4064-a532-12949bfaaba2-tempo-data-science-tempomonolithic-mtls" pod "85273358-99d0-4064-a532-12949bfaaba2" (UID: "85273358-99d0-4064-a532-12949bfaaba2"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:30.900316 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:30.900280 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85273358-99d0-4064-a532-12949bfaaba2-gateway-tenants" pod "85273358-99d0-4064-a532-12949bfaaba2" (UID: "85273358-99d0-4064-a532-12949bfaaba2"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:30.900845 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:30.900828 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062" pod "85273358-99d0-4064-a532-12949bfaaba2" (UID: "85273358-99d0-4064-a532-12949bfaaba2"). InnerVolumeSpecName "pvc-4491ac70-4597-43a7-a792-f0813314e433". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:12:30.998583 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:30.998555 5274 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/85273358-99d0-4064-a532-12949bfaaba2-gateway-rbac\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:30.998583 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:30.998580 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/85273358-99d0-4064-a532-12949bfaaba2-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:30.998742 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:30.998620 5274 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" " Oct 31 00:12:30.998742 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:30.998636 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/85273358-99d0-4064-a532-12949bfaaba2-tempo-conf\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:30.998742 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:30.998650 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85273358-99d0-4064-a532-12949bfaaba2-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:30.998742 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:30.998665 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7f5c\" (UniqueName: \"kubernetes.io/projected/85273358-99d0-4064-a532-12949bfaaba2-kube-api-access-m7f5c\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:30.998742 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:30.998679 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/85273358-99d0-4064-a532-12949bfaaba2-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:30.998742 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:30.998693 5274 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/85273358-99d0-4064-a532-12949bfaaba2-gateway-tenants\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:31.013196 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:31.013179 5274 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-4491ac70-4597-43a7-a792-f0813314e433" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062") on node "ip-10-0-129-70.us-east-2.compute.internal" Oct 31 00:12:31.100013 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:31.099979 5274 reconciler_common.go:299] "Volume detached for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"csi-9ce3c80b5d5deac9dfdb953db4fb017080b654a85db76a2937e67cd541d89057\"" Oct 31 00:12:31.338363 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:31.338281 5274 generic.go:356] "Generic (PLEG): container finished" podID="85273358-99d0-4064-a532-12949bfaaba2" containerID="044d0657054494dab71a5b9edb54f4c5923ab1a156a00129ddff6ab398d47cc4" exitCode=0 Oct 31 00:12:31.338748 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:31.338367 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:31.338748 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:31.338371 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"85273358-99d0-4064-a532-12949bfaaba2","Type":"ContainerDied","Data":"044d0657054494dab71a5b9edb54f4c5923ab1a156a00129ddff6ab398d47cc4"} Oct 31 00:12:31.338748 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:31.338471 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"85273358-99d0-4064-a532-12949bfaaba2","Type":"ContainerDied","Data":"71093101977e0ca31ca4fb7556f2c8e28c6b1510edaa8edb58240554707f23ff"} Oct 31 00:12:31.338748 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:31.338531 5274 scope.go:122] "RemoveContainer" containerID="10342dce14b9bf67d3af221eb2570fe11675463ea19e10c7ef72b38153092fa5" Oct 31 00:12:31.349765 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:31.349750 5274 scope.go:122] "RemoveContainer" containerID="044d0657054494dab71a5b9edb54f4c5923ab1a156a00129ddff6ab398d47cc4" Oct 31 00:12:31.358461 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:31.358444 5274 scope.go:122] "RemoveContainer" containerID="945282e431dad2b50fa915f70efe02dd3ce966db0506e04a93eb7c4cb52679e8" Oct 31 00:12:31.363147 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:31.363122 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:12:31.366603 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:31.366583 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:12:31.368071 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:31.367987 5274 scope.go:122] "RemoveContainer" containerID="10342dce14b9bf67d3af221eb2570fe11675463ea19e10c7ef72b38153092fa5" Oct 31 00:12:31.368441 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:12:31.368413 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10342dce14b9bf67d3af221eb2570fe11675463ea19e10c7ef72b38153092fa5\": container with ID starting with 10342dce14b9bf67d3af221eb2570fe11675463ea19e10c7ef72b38153092fa5 not found: ID does not exist" containerID="10342dce14b9bf67d3af221eb2570fe11675463ea19e10c7ef72b38153092fa5" Oct 31 00:12:31.368537 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:31.368451 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10342dce14b9bf67d3af221eb2570fe11675463ea19e10c7ef72b38153092fa5"} err="failed to get container status \"10342dce14b9bf67d3af221eb2570fe11675463ea19e10c7ef72b38153092fa5\": rpc error: code = NotFound desc = could not find container \"10342dce14b9bf67d3af221eb2570fe11675463ea19e10c7ef72b38153092fa5\": container with ID starting with 10342dce14b9bf67d3af221eb2570fe11675463ea19e10c7ef72b38153092fa5 not found: ID does not exist" Oct 31 00:12:31.368537 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:31.368475 5274 scope.go:122] "RemoveContainer" containerID="044d0657054494dab71a5b9edb54f4c5923ab1a156a00129ddff6ab398d47cc4" Oct 31 00:12:31.368902 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:12:31.368808 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"044d0657054494dab71a5b9edb54f4c5923ab1a156a00129ddff6ab398d47cc4\": container with ID starting with 044d0657054494dab71a5b9edb54f4c5923ab1a156a00129ddff6ab398d47cc4 not found: ID does not exist" containerID="044d0657054494dab71a5b9edb54f4c5923ab1a156a00129ddff6ab398d47cc4" Oct 31 00:12:31.368902 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:31.368837 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"044d0657054494dab71a5b9edb54f4c5923ab1a156a00129ddff6ab398d47cc4"} err="failed to get container status \"044d0657054494dab71a5b9edb54f4c5923ab1a156a00129ddff6ab398d47cc4\": rpc error: code = NotFound desc = could not find container \"044d0657054494dab71a5b9edb54f4c5923ab1a156a00129ddff6ab398d47cc4\": container with ID starting with 044d0657054494dab71a5b9edb54f4c5923ab1a156a00129ddff6ab398d47cc4 not found: ID does not exist" Oct 31 00:12:31.368902 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:31.368853 5274 scope.go:122] "RemoveContainer" containerID="945282e431dad2b50fa915f70efe02dd3ce966db0506e04a93eb7c4cb52679e8" Oct 31 00:12:31.369371 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:12:31.369350 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"945282e431dad2b50fa915f70efe02dd3ce966db0506e04a93eb7c4cb52679e8\": container with ID starting with 945282e431dad2b50fa915f70efe02dd3ce966db0506e04a93eb7c4cb52679e8 not found: ID does not exist" containerID="945282e431dad2b50fa915f70efe02dd3ce966db0506e04a93eb7c4cb52679e8" Oct 31 00:12:31.369456 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:31.369380 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"945282e431dad2b50fa915f70efe02dd3ce966db0506e04a93eb7c4cb52679e8"} err="failed to get container status \"945282e431dad2b50fa915f70efe02dd3ce966db0506e04a93eb7c4cb52679e8\": rpc error: code = NotFound desc = could not find container \"945282e431dad2b50fa915f70efe02dd3ce966db0506e04a93eb7c4cb52679e8\": container with ID starting with 945282e431dad2b50fa915f70efe02dd3ce966db0506e04a93eb7c4cb52679e8 not found: ID does not exist" Oct 31 00:12:31.482468 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:31.482437 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85273358-99d0-4064-a532-12949bfaaba2" path="/var/lib/kubelet/pods/85273358-99d0-4064-a532-12949bfaaba2/volumes" Oct 31 00:12:41.340691 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:41.340661 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-fjl92" Oct 31 00:12:41.404415 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:41.404384 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-fjl92"] Oct 31 00:12:41.404799 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:41.404561 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-fjl92" podUID="122ca78b-1834-4ae8-abb1-7962c7f5797d" containerName="korrel8r" containerID="cri-o://4eeed830b5fa1173184e535bf51d897be0033f494d163f82d4753798164a7cea" gracePeriod=30 Oct 31 00:12:41.652092 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:41.652070 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-fjl92" Oct 31 00:12:41.683998 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:41.683951 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/122ca78b-1834-4ae8-abb1-7962c7f5797d-config\" (UniqueName: \"kubernetes.io/configmap/122ca78b-1834-4ae8-abb1-7962c7f5797d-config\") pod \"122ca78b-1834-4ae8-abb1-7962c7f5797d\" (UID: \"122ca78b-1834-4ae8-abb1-7962c7f5797d\") " Oct 31 00:12:41.684149 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:41.684077 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/122ca78b-1834-4ae8-abb1-7962c7f5797d-tls\" (UniqueName: \"kubernetes.io/secret/122ca78b-1834-4ae8-abb1-7962c7f5797d-tls\") pod \"122ca78b-1834-4ae8-abb1-7962c7f5797d\" (UID: \"122ca78b-1834-4ae8-abb1-7962c7f5797d\") " Oct 31 00:12:41.684149 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:41.684120 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/122ca78b-1834-4ae8-abb1-7962c7f5797d-kube-api-access-dwldz\" (UniqueName: \"kubernetes.io/projected/122ca78b-1834-4ae8-abb1-7962c7f5797d-kube-api-access-dwldz\") pod \"122ca78b-1834-4ae8-abb1-7962c7f5797d\" (UID: \"122ca78b-1834-4ae8-abb1-7962c7f5797d\") " Oct 31 00:12:41.684302 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:41.684276 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/122ca78b-1834-4ae8-abb1-7962c7f5797d-config" pod "122ca78b-1834-4ae8-abb1-7962c7f5797d" (UID: "122ca78b-1834-4ae8-abb1-7962c7f5797d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:12:41.684376 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:41.684363 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/122ca78b-1834-4ae8-abb1-7962c7f5797d-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:41.686153 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:41.686127 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/122ca78b-1834-4ae8-abb1-7962c7f5797d-kube-api-access-dwldz" pod "122ca78b-1834-4ae8-abb1-7962c7f5797d" (UID: "122ca78b-1834-4ae8-abb1-7962c7f5797d"). InnerVolumeSpecName "kube-api-access-dwldz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:12:41.686153 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:41.686138 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/122ca78b-1834-4ae8-abb1-7962c7f5797d-tls" pod "122ca78b-1834-4ae8-abb1-7962c7f5797d" (UID: "122ca78b-1834-4ae8-abb1-7962c7f5797d"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:41.785306 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:41.785281 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/122ca78b-1834-4ae8-abb1-7962c7f5797d-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:41.785306 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:41.785308 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwldz\" (UniqueName: \"kubernetes.io/projected/122ca78b-1834-4ae8-abb1-7962c7f5797d-kube-api-access-dwldz\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:42.396025 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:42.395984 5274 generic.go:356] "Generic (PLEG): container finished" podID="122ca78b-1834-4ae8-abb1-7962c7f5797d" containerID="4eeed830b5fa1173184e535bf51d897be0033f494d163f82d4753798164a7cea" exitCode=0 Oct 31 00:12:42.396415 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:42.396042 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-fjl92" Oct 31 00:12:42.396415 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:42.396048 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-fjl92" event={"ID":"122ca78b-1834-4ae8-abb1-7962c7f5797d","Type":"ContainerDied","Data":"4eeed830b5fa1173184e535bf51d897be0033f494d163f82d4753798164a7cea"} Oct 31 00:12:42.396415 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:42.396144 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-fjl92" event={"ID":"122ca78b-1834-4ae8-abb1-7962c7f5797d","Type":"ContainerDied","Data":"8b8f4095329d3b4052ecf837975145852a7751d438839a013834509a9b69f399"} Oct 31 00:12:42.396415 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:42.396177 5274 scope.go:122] "RemoveContainer" containerID="4eeed830b5fa1173184e535bf51d897be0033f494d163f82d4753798164a7cea" Oct 31 00:12:42.405886 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:42.405764 5274 scope.go:122] "RemoveContainer" containerID="4eeed830b5fa1173184e535bf51d897be0033f494d163f82d4753798164a7cea" Oct 31 00:12:42.406078 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:12:42.406058 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eeed830b5fa1173184e535bf51d897be0033f494d163f82d4753798164a7cea\": container with ID starting with 4eeed830b5fa1173184e535bf51d897be0033f494d163f82d4753798164a7cea not found: ID does not exist" containerID="4eeed830b5fa1173184e535bf51d897be0033f494d163f82d4753798164a7cea" Oct 31 00:12:42.406183 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:42.406086 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eeed830b5fa1173184e535bf51d897be0033f494d163f82d4753798164a7cea"} err="failed to get container status \"4eeed830b5fa1173184e535bf51d897be0033f494d163f82d4753798164a7cea\": rpc error: code = NotFound desc = could not find container \"4eeed830b5fa1173184e535bf51d897be0033f494d163f82d4753798164a7cea\": container with ID starting with 4eeed830b5fa1173184e535bf51d897be0033f494d163f82d4753798164a7cea not found: ID does not exist" Oct 31 00:12:42.419913 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:42.419887 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-fjl92"] Oct 31 00:12:42.424314 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:42.424293 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-fjl92"] Oct 31 00:12:43.115836 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.115806 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:12:43.116243 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.116228 5274 cpu_manager.go:414] "containerMap: removing container" podUID="85273358-99d0-4064-a532-12949bfaaba2" containerName="tempo" Oct 31 00:12:43.116296 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.116244 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85273358-99d0-4064-a532-12949bfaaba2" containerName="tempo" Oct 31 00:12:43.116296 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.116264 5274 cpu_manager.go:414] "containerMap: removing container" podUID="85273358-99d0-4064-a532-12949bfaaba2" containerName="tempo-gateway" Oct 31 00:12:43.116296 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.116269 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85273358-99d0-4064-a532-12949bfaaba2" containerName="tempo-gateway" Oct 31 00:12:43.116296 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.116280 5274 cpu_manager.go:414] "containerMap: removing container" podUID="122ca78b-1834-4ae8-abb1-7962c7f5797d" containerName="korrel8r" Oct 31 00:12:43.116296 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.116285 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="122ca78b-1834-4ae8-abb1-7962c7f5797d" containerName="korrel8r" Oct 31 00:12:43.116296 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.116294 5274 cpu_manager.go:414] "containerMap: removing container" podUID="85273358-99d0-4064-a532-12949bfaaba2" containerName="tempo-gateway-opa" Oct 31 00:12:43.116520 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.116299 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85273358-99d0-4064-a532-12949bfaaba2" containerName="tempo-gateway-opa" Oct 31 00:12:43.116520 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.116351 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="85273358-99d0-4064-a532-12949bfaaba2" containerName="tempo-gateway" Oct 31 00:12:43.116520 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.116360 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="122ca78b-1834-4ae8-abb1-7962c7f5797d" containerName="korrel8r" Oct 31 00:12:43.116520 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.116367 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="85273358-99d0-4064-a532-12949bfaaba2" containerName="tempo-gateway-opa" Oct 31 00:12:43.116520 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.116376 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="85273358-99d0-4064-a532-12949bfaaba2" containerName="tempo" Oct 31 00:12:43.121428 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.121408 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:43.125202 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.125179 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\"" Oct 31 00:12:43.125585 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.125568 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 31 00:12:43.125585 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.125573 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 31 00:12:43.126111 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.126092 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\"" Oct 31 00:12:43.126198 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.126150 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\"" Oct 31 00:12:43.126198 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.126170 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-wt4gn\"" Oct 31 00:12:43.126198 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.126189 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\"" Oct 31 00:12:43.126341 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.126218 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\"" Oct 31 00:12:43.134550 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.134530 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:12:43.195479 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.195447 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:43.195614 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.195493 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:43.195668 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.195603 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:43.195711 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.195664 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:43.195762 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.195744 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:43.195808 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.195778 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:43.195909 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.195892 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:43.195999 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.195923 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qhtn\" (UniqueName: \"kubernetes.io/projected/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-kube-api-access-2qhtn\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:43.297327 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.297300 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:43.297459 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.297331 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:43.297459 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.297361 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:43.297459 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.297379 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qhtn\" (UniqueName: \"kubernetes.io/projected/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-kube-api-access-2qhtn\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:43.297630 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:12:43.297465 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway-serving-cert: secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found Oct 31 00:12:43.297630 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.297498 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:43.297630 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:12:43.297535 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-tempo-data-science-tempomonolithic-gateway-serving-cert podName:1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6 nodeName:}" failed. No retries permitted until 2026-10-31 00:12:43.797513883 +0000 UTC m=+800.434332577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-gateway-serving-cert" (UniqueName: "kubernetes.io/secret/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-tempo-data-science-tempomonolithic-gateway-serving-cert") pod "tempo-data-science-tempomonolithic-0" (UID: "1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6") : secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found Oct 31 00:12:43.297630 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.297583 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:43.297845 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.297629 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:43.297845 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.297674 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:43.298197 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.298166 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:43.298197 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.298191 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:43.298361 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.298263 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:43.298361 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.298275 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:43.299948 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.299923 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:43.300055 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.300017 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:43.309594 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.309574 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qhtn\" (UniqueName: \"kubernetes.io/projected/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-kube-api-access-2qhtn\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:43.481012 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.480894 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="122ca78b-1834-4ae8-abb1-7962c7f5797d" path="/var/lib/kubelet/pods/122ca78b-1834-4ae8-abb1-7962c7f5797d/volumes" Oct 31 00:12:43.803229 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.803151 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:43.805595 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:43.805573 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:49.953341 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:49.953305 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:49.961285 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:49.961262 5274 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:50.054572 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:50.054545 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:50.054694 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:50.054674 5274 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:50.058240 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:50.058226 5274 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") DevicePath \"csi-9ce3c80b5d5deac9dfdb953db4fb017080b654a85db76a2937e67cd541d89057\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:50.147404 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:50.147371 5274 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/151619720cb39d40781bf258206fb0be746b77e5f55879e189e4cf0d6e46dfb0/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:50.163028 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:50.163008 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:50.339710 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:50.339652 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:50.621137 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:50.621108 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:12:50.622012 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:12:50.621988 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f3caa74_9f4b_4e2e_a3c5_a6305658a2f6.slice/crio-4985d776aa22533019283b02468e9e0df16ed4b3388d1aea1d5d9469264d9684 WatchSource:0}: Error finding container 4985d776aa22533019283b02468e9e0df16ed4b3388d1aea1d5d9469264d9684: Status 404 returned error can't find the container with id 4985d776aa22533019283b02468e9e0df16ed4b3388d1aea1d5d9469264d9684 Oct 31 00:12:50.897115 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:12:50.897044 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:12:51.446369 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:51.446334 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6","Type":"ContainerStarted","Data":"32de2e3a605838ada6a60dd148a7394db61b60cc424a4f80acdb91696de7b856"} Oct 31 00:12:51.446796 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:51.446397 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6","Type":"ContainerStarted","Data":"6e2fe3eff41731d269aaca74135ee3030bcfab9e0d199ee65c6e311d55160645"} Oct 31 00:12:51.446796 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:51.446429 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:51.446796 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:51.446453 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6","Type":"ContainerStarted","Data":"0b74843716c61c21c4c731a08e5213f1e711b7456e47967191d2021e00e519d9"} Oct 31 00:12:51.446796 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:51.446473 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6","Type":"ContainerStarted","Data":"4985d776aa22533019283b02468e9e0df16ed4b3388d1aea1d5d9469264d9684"} Oct 31 00:12:51.446796 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:51.446497 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:51.448968 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:51.448935 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:51.454726 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:51.454709 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:51.474326 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:51.474261 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=8.474247441 podStartE2EDuration="8.474247441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:12:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:51.473039898 +0000 UTC m=+808.109858615" watchObservedRunningTime="2026-10-31 00:12:51.474247441 +0000 UTC m=+808.111066157" Oct 31 00:12:52.452371 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:52.452343 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:53.613280 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:53.613246 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-8jrpk"] Oct 31 00:12:53.616932 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:53.616913 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8jrpk" Oct 31 00:12:53.627264 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:53.627242 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-8jrpk"] Oct 31 00:12:53.686790 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:53.686765 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb-config\") pod \"korrel8r-5cbcc89b8f-8jrpk\" (UID: \"02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8jrpk" Oct 31 00:12:53.686905 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:53.686814 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb-tls\") pod \"korrel8r-5cbcc89b8f-8jrpk\" (UID: \"02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8jrpk" Oct 31 00:12:53.686976 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:53.686912 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7b7g\" (UniqueName: \"kubernetes.io/projected/02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb-kube-api-access-t7b7g\") pod \"korrel8r-5cbcc89b8f-8jrpk\" (UID: \"02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8jrpk" Oct 31 00:12:53.787413 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:53.787379 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb-config\") pod \"korrel8r-5cbcc89b8f-8jrpk\" (UID: \"02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8jrpk" Oct 31 00:12:53.787595 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:53.787452 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb-tls\") pod \"korrel8r-5cbcc89b8f-8jrpk\" (UID: \"02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8jrpk" Oct 31 00:12:53.787595 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:53.787524 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7b7g\" (UniqueName: \"kubernetes.io/projected/02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb-kube-api-access-t7b7g\") pod \"korrel8r-5cbcc89b8f-8jrpk\" (UID: \"02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8jrpk" Oct 31 00:12:53.788415 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:53.788393 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb-config\") pod \"korrel8r-5cbcc89b8f-8jrpk\" (UID: \"02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8jrpk" Oct 31 00:12:53.790136 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:53.790112 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb-tls\") pod \"korrel8r-5cbcc89b8f-8jrpk\" (UID: \"02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8jrpk" Oct 31 00:12:53.800321 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:53.800299 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7b7g\" (UniqueName: \"kubernetes.io/projected/02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb-kube-api-access-t7b7g\") pod \"korrel8r-5cbcc89b8f-8jrpk\" (UID: \"02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8jrpk" Oct 31 00:12:53.931480 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:53.931448 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8jrpk" Oct 31 00:12:54.217468 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:54.217399 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-8jrpk"] Oct 31 00:12:54.219846 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:12:54.219817 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02bd33f0_a57a_4c9f_bc4d_7e2e8e5ebafb.slice/crio-3e64fbd14c3864ba692a9cfc52a2b20c37b221c5261e85d7a86b14c56b631146 WatchSource:0}: Error finding container 3e64fbd14c3864ba692a9cfc52a2b20c37b221c5261e85d7a86b14c56b631146: Status 404 returned error can't find the container with id 3e64fbd14c3864ba692a9cfc52a2b20c37b221c5261e85d7a86b14c56b631146 Oct 31 00:12:54.463344 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:54.463304 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8jrpk" event={"ID":"02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb","Type":"ContainerStarted","Data":"6ef3559a114f12d1f4aacc1175c13b7738775c3f666c9542c526c43fd0f7afbe"} Oct 31 00:12:54.463553 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:54.463361 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8jrpk" event={"ID":"02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb","Type":"ContainerStarted","Data":"3e64fbd14c3864ba692a9cfc52a2b20c37b221c5261e85d7a86b14c56b631146"} Oct 31 00:12:54.463553 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:54.463382 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8jrpk" Oct 31 00:12:54.485026 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:54.484904 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8jrpk" podStartSLOduration=1.484885961 podStartE2EDuration="1.484885961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:54.483981244 +0000 UTC m=+811.120799961" watchObservedRunningTime="2026-10-31 00:12:54.484885961 +0000 UTC m=+811.121708217" Oct 31 00:12:54.516824 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:54.516789 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:12:54.517075 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:54.517046 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6" containerName="tempo" containerID="cri-o://0b74843716c61c21c4c731a08e5213f1e711b7456e47967191d2021e00e519d9" gracePeriod=30 Oct 31 00:12:54.517227 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:54.517081 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6" containerName="tempo-gateway-opa" containerID="cri-o://32de2e3a605838ada6a60dd148a7394db61b60cc424a4f80acdb91696de7b856" gracePeriod=30 Oct 31 00:12:54.517227 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:54.517081 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6" containerName="tempo-gateway" containerID="cri-o://6e2fe3eff41731d269aaca74135ee3030bcfab9e0d199ee65c6e311d55160645" gracePeriod=30 Oct 31 00:12:55.471156 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:55.471124 5274 generic.go:356] "Generic (PLEG): container finished" podID="1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6" containerID="32de2e3a605838ada6a60dd148a7394db61b60cc424a4f80acdb91696de7b856" exitCode=0 Oct 31 00:12:55.471156 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:55.471147 5274 generic.go:356] "Generic (PLEG): container finished" podID="1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6" containerID="0b74843716c61c21c4c731a08e5213f1e711b7456e47967191d2021e00e519d9" exitCode=0 Oct 31 00:12:55.471611 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:55.471198 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6","Type":"ContainerDied","Data":"32de2e3a605838ada6a60dd148a7394db61b60cc424a4f80acdb91696de7b856"} Oct 31 00:12:55.471611 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:55.471264 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6","Type":"ContainerDied","Data":"0b74843716c61c21c4c731a08e5213f1e711b7456e47967191d2021e00e519d9"} Oct 31 00:12:55.773099 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:55.773075 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:55.808443 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:55.808419 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-tempo-data-science-tempomonolithic-ca-bundle\") pod \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") " Oct 31 00:12:55.808570 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:55.808467 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") " Oct 31 00:12:55.808570 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:55.808491 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-kube-api-access-2qhtn\" (UniqueName: \"kubernetes.io/projected/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-kube-api-access-2qhtn\") pod \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") " Oct 31 00:12:55.808740 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:55.808721 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") " Oct 31 00:12:55.808790 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:55.808777 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-gateway-rbac\") pod \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") " Oct 31 00:12:55.808843 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:55.808799 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-tempo-data-science-tempomonolithic-ca-bundle" pod "1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6" (UID: "1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:12:55.808902 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:55.808883 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-tempo-conf\") pod \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") " Oct 31 00:12:55.808967 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:55.808941 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-gateway-tenants\") pod \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") " Oct 31 00:12:55.809429 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:55.809087 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") " Oct 31 00:12:55.809429 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:55.809119 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-gateway-rbac" pod "1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6" (UID: "1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:12:55.809429 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:55.809125 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-tempo-data-science-tempomonolithic-mtls\") pod \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\" (UID: \"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6\") " Oct 31 00:12:55.809429 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:55.809322 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-tempo-conf" pod "1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6" (UID: "1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:12:55.809429 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:55.809410 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:55.809429 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:55.809430 5274 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-gateway-rbac\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:55.809429 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:55.809444 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-tempo-conf\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:55.809860 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:55.809448 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-tempo-data-science-tempomonolithic-serving-cabundle" pod "1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6" (UID: "1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:12:55.811223 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:55.811031 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-kube-api-access-2qhtn" pod "1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6" (UID: "1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6"). InnerVolumeSpecName "kube-api-access-2qhtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:12:55.811330 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:55.811268 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6" (UID: "1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:55.811395 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:55.811330 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-gateway-tenants" pod "1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6" (UID: "1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:55.811618 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:55.811594 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-tempo-data-science-tempomonolithic-mtls" pod "1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6" (UID: "1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:55.812236 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:55.812208 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062" pod "1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6" (UID: "1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6"). InnerVolumeSpecName "pvc-4491ac70-4597-43a7-a792-f0813314e433". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:12:55.910477 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:55.910443 5274 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-gateway-tenants\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:55.910477 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:55.910470 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:55.910477 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:55.910482 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:55.910701 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:55.910493 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:55.910701 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:55.910503 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qhtn\" (UniqueName: \"kubernetes.io/projected/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6-kube-api-access-2qhtn\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:55.910701 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:55.910530 5274 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" " Oct 31 00:12:55.924109 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:55.924086 5274 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-4491ac70-4597-43a7-a792-f0813314e433" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062") on node "ip-10-0-129-70.us-east-2.compute.internal" Oct 31 00:12:56.011938 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:56.011842 5274 reconciler_common.go:299] "Volume detached for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"csi-9ce3c80b5d5deac9dfdb953db4fb017080b654a85db76a2937e67cd541d89057\"" Oct 31 00:12:56.478627 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:56.478587 5274 generic.go:356] "Generic (PLEG): container finished" podID="1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6" containerID="6e2fe3eff41731d269aaca74135ee3030bcfab9e0d199ee65c6e311d55160645" exitCode=0 Oct 31 00:12:56.479105 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:56.478627 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6","Type":"ContainerDied","Data":"6e2fe3eff41731d269aaca74135ee3030bcfab9e0d199ee65c6e311d55160645"} Oct 31 00:12:56.479105 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:56.478650 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:12:56.479105 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:56.478688 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6","Type":"ContainerDied","Data":"4985d776aa22533019283b02468e9e0df16ed4b3388d1aea1d5d9469264d9684"} Oct 31 00:12:56.479105 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:56.478723 5274 scope.go:122] "RemoveContainer" containerID="32de2e3a605838ada6a60dd148a7394db61b60cc424a4f80acdb91696de7b856" Oct 31 00:12:56.489233 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:56.489154 5274 scope.go:122] "RemoveContainer" containerID="6e2fe3eff41731d269aaca74135ee3030bcfab9e0d199ee65c6e311d55160645" Oct 31 00:12:56.498312 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:56.498298 5274 scope.go:122] "RemoveContainer" containerID="0b74843716c61c21c4c731a08e5213f1e711b7456e47967191d2021e00e519d9" Oct 31 00:12:56.503318 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:56.503298 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:12:56.506615 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:56.506593 5274 scope.go:122] "RemoveContainer" containerID="32de2e3a605838ada6a60dd148a7394db61b60cc424a4f80acdb91696de7b856" Oct 31 00:12:56.506852 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:12:56.506833 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32de2e3a605838ada6a60dd148a7394db61b60cc424a4f80acdb91696de7b856\": container with ID starting with 32de2e3a605838ada6a60dd148a7394db61b60cc424a4f80acdb91696de7b856 not found: ID does not exist" containerID="32de2e3a605838ada6a60dd148a7394db61b60cc424a4f80acdb91696de7b856" Oct 31 00:12:56.506932 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:56.506865 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32de2e3a605838ada6a60dd148a7394db61b60cc424a4f80acdb91696de7b856"} err="failed to get container status \"32de2e3a605838ada6a60dd148a7394db61b60cc424a4f80acdb91696de7b856\": rpc error: code = NotFound desc = could not find container \"32de2e3a605838ada6a60dd148a7394db61b60cc424a4f80acdb91696de7b856\": container with ID starting with 32de2e3a605838ada6a60dd148a7394db61b60cc424a4f80acdb91696de7b856 not found: ID does not exist" Oct 31 00:12:56.506932 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:56.506889 5274 scope.go:122] "RemoveContainer" containerID="6e2fe3eff41731d269aaca74135ee3030bcfab9e0d199ee65c6e311d55160645" Oct 31 00:12:56.507155 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:12:56.507138 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e2fe3eff41731d269aaca74135ee3030bcfab9e0d199ee65c6e311d55160645\": container with ID starting with 6e2fe3eff41731d269aaca74135ee3030bcfab9e0d199ee65c6e311d55160645 not found: ID does not exist" containerID="6e2fe3eff41731d269aaca74135ee3030bcfab9e0d199ee65c6e311d55160645" Oct 31 00:12:56.507221 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:56.507165 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e2fe3eff41731d269aaca74135ee3030bcfab9e0d199ee65c6e311d55160645"} err="failed to get container status \"6e2fe3eff41731d269aaca74135ee3030bcfab9e0d199ee65c6e311d55160645\": rpc error: code = NotFound desc = could not find container \"6e2fe3eff41731d269aaca74135ee3030bcfab9e0d199ee65c6e311d55160645\": container with ID starting with 6e2fe3eff41731d269aaca74135ee3030bcfab9e0d199ee65c6e311d55160645 not found: ID does not exist" Oct 31 00:12:56.507221 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:56.507185 5274 scope.go:122] "RemoveContainer" containerID="0b74843716c61c21c4c731a08e5213f1e711b7456e47967191d2021e00e519d9" Oct 31 00:12:56.507430 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:12:56.507408 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b74843716c61c21c4c731a08e5213f1e711b7456e47967191d2021e00e519d9\": container with ID starting with 0b74843716c61c21c4c731a08e5213f1e711b7456e47967191d2021e00e519d9 not found: ID does not exist" containerID="0b74843716c61c21c4c731a08e5213f1e711b7456e47967191d2021e00e519d9" Oct 31 00:12:56.507482 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:56.507440 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b74843716c61c21c4c731a08e5213f1e711b7456e47967191d2021e00e519d9"} err="failed to get container status \"0b74843716c61c21c4c731a08e5213f1e711b7456e47967191d2021e00e519d9\": rpc error: code = NotFound desc = could not find container \"0b74843716c61c21c4c731a08e5213f1e711b7456e47967191d2021e00e519d9\": container with ID starting with 0b74843716c61c21c4c731a08e5213f1e711b7456e47967191d2021e00e519d9 not found: ID does not exist" Oct 31 00:12:56.512036 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:56.512016 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:12:57.481292 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:12:57.481252 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6" path="/var/lib/kubelet/pods/1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6/volumes" Oct 31 00:13:05.473586 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:05.473556 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8jrpk" Oct 31 00:13:05.522083 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:05.522048 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-24bxz"] Oct 31 00:13:05.522340 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:05.522292 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-24bxz" podUID="4c2b4fae-0239-488e-bfc0-b11500d74773" containerName="korrel8r" containerID="cri-o://acd428f423c2f6d58e06de71d02e95255ae880981c6b789e6016c4dc9c6fb27b" gracePeriod=30 Oct 31 00:13:05.768634 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:05.768609 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-24bxz" Oct 31 00:13:05.896624 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:05.896552 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c2b4fae-0239-488e-bfc0-b11500d74773-kube-api-access-hqw8l\" (UniqueName: \"kubernetes.io/projected/4c2b4fae-0239-488e-bfc0-b11500d74773-kube-api-access-hqw8l\") pod \"4c2b4fae-0239-488e-bfc0-b11500d74773\" (UID: \"4c2b4fae-0239-488e-bfc0-b11500d74773\") " Oct 31 00:13:05.896624 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:05.896594 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c2b4fae-0239-488e-bfc0-b11500d74773-config\" (UniqueName: \"kubernetes.io/configmap/4c2b4fae-0239-488e-bfc0-b11500d74773-config\") pod \"4c2b4fae-0239-488e-bfc0-b11500d74773\" (UID: \"4c2b4fae-0239-488e-bfc0-b11500d74773\") " Oct 31 00:13:05.896818 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:05.896638 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c2b4fae-0239-488e-bfc0-b11500d74773-tls\" (UniqueName: \"kubernetes.io/secret/4c2b4fae-0239-488e-bfc0-b11500d74773-tls\") pod \"4c2b4fae-0239-488e-bfc0-b11500d74773\" (UID: \"4c2b4fae-0239-488e-bfc0-b11500d74773\") " Oct 31 00:13:05.896997 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:05.896974 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c2b4fae-0239-488e-bfc0-b11500d74773-config" pod "4c2b4fae-0239-488e-bfc0-b11500d74773" (UID: "4c2b4fae-0239-488e-bfc0-b11500d74773"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:13:05.898732 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:05.898702 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c2b4fae-0239-488e-bfc0-b11500d74773-kube-api-access-hqw8l" pod "4c2b4fae-0239-488e-bfc0-b11500d74773" (UID: "4c2b4fae-0239-488e-bfc0-b11500d74773"). InnerVolumeSpecName "kube-api-access-hqw8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:13:05.898844 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:05.898785 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c2b4fae-0239-488e-bfc0-b11500d74773-tls" pod "4c2b4fae-0239-488e-bfc0-b11500d74773" (UID: "4c2b4fae-0239-488e-bfc0-b11500d74773"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:05.997839 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:05.997759 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqw8l\" (UniqueName: \"kubernetes.io/projected/4c2b4fae-0239-488e-bfc0-b11500d74773-kube-api-access-hqw8l\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:05.997839 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:05.997787 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4c2b4fae-0239-488e-bfc0-b11500d74773-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:05.997839 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:05.997796 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/4c2b4fae-0239-488e-bfc0-b11500d74773-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:06.532305 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:06.532275 5274 generic.go:356] "Generic (PLEG): container finished" podID="4c2b4fae-0239-488e-bfc0-b11500d74773" containerID="acd428f423c2f6d58e06de71d02e95255ae880981c6b789e6016c4dc9c6fb27b" exitCode=0 Oct 31 00:13:06.532703 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:06.532317 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-24bxz" Oct 31 00:13:06.532703 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:06.532318 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-24bxz" event={"ID":"4c2b4fae-0239-488e-bfc0-b11500d74773","Type":"ContainerDied","Data":"acd428f423c2f6d58e06de71d02e95255ae880981c6b789e6016c4dc9c6fb27b"} Oct 31 00:13:06.532703 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:06.532440 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-24bxz" event={"ID":"4c2b4fae-0239-488e-bfc0-b11500d74773","Type":"ContainerDied","Data":"798a364bfa9ece85b2e3238dcb00a8ff8179638a0f262427ae3ecbe73cf9b01a"} Oct 31 00:13:06.532703 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:06.532467 5274 scope.go:122] "RemoveContainer" containerID="acd428f423c2f6d58e06de71d02e95255ae880981c6b789e6016c4dc9c6fb27b" Oct 31 00:13:06.543741 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:06.543723 5274 scope.go:122] "RemoveContainer" containerID="acd428f423c2f6d58e06de71d02e95255ae880981c6b789e6016c4dc9c6fb27b" Oct 31 00:13:06.544017 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:06.543999 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acd428f423c2f6d58e06de71d02e95255ae880981c6b789e6016c4dc9c6fb27b\": container with ID starting with acd428f423c2f6d58e06de71d02e95255ae880981c6b789e6016c4dc9c6fb27b not found: ID does not exist" containerID="acd428f423c2f6d58e06de71d02e95255ae880981c6b789e6016c4dc9c6fb27b" Oct 31 00:13:06.544065 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:06.544028 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acd428f423c2f6d58e06de71d02e95255ae880981c6b789e6016c4dc9c6fb27b"} err="failed to get container status \"acd428f423c2f6d58e06de71d02e95255ae880981c6b789e6016c4dc9c6fb27b\": rpc error: code = NotFound desc = could not find container \"acd428f423c2f6d58e06de71d02e95255ae880981c6b789e6016c4dc9c6fb27b\": container with ID starting with acd428f423c2f6d58e06de71d02e95255ae880981c6b789e6016c4dc9c6fb27b not found: ID does not exist" Oct 31 00:13:06.557333 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:06.557313 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-24bxz"] Oct 31 00:13:06.561417 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:06.561389 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-24bxz"] Oct 31 00:13:07.481174 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:07.481146 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c2b4fae-0239-488e-bfc0-b11500d74773" path="/var/lib/kubelet/pods/4c2b4fae-0239-488e-bfc0-b11500d74773/volumes" Oct 31 00:13:20.898746 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:20.898712 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:13:30.425093 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:30.425057 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-66pkr"] Oct 31 00:13:30.425546 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:30.425426 5274 cpu_manager.go:414] "containerMap: removing container" podUID="1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6" containerName="tempo-gateway-opa" Oct 31 00:13:30.425546 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:30.425436 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6" containerName="tempo-gateway-opa" Oct 31 00:13:30.425546 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:30.425465 5274 cpu_manager.go:414] "containerMap: removing container" podUID="1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6" containerName="tempo" Oct 31 00:13:30.425546 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:30.425470 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6" containerName="tempo" Oct 31 00:13:30.425546 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:30.425487 5274 cpu_manager.go:414] "containerMap: removing container" podUID="4c2b4fae-0239-488e-bfc0-b11500d74773" containerName="korrel8r" Oct 31 00:13:30.425546 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:30.425493 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c2b4fae-0239-488e-bfc0-b11500d74773" containerName="korrel8r" Oct 31 00:13:30.425546 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:30.425501 5274 cpu_manager.go:414] "containerMap: removing container" podUID="1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6" containerName="tempo-gateway" Oct 31 00:13:30.425546 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:30.425506 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6" containerName="tempo-gateway" Oct 31 00:13:30.425778 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:30.425559 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6" containerName="tempo" Oct 31 00:13:30.425778 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:30.425567 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c2b4fae-0239-488e-bfc0-b11500d74773" containerName="korrel8r" Oct 31 00:13:30.425778 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:30.425574 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6" containerName="tempo-gateway" Oct 31 00:13:30.425778 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:30.425580 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f3caa74-9f4b-4e2e-a3c5-a6305658a2f6" containerName="tempo-gateway-opa" Oct 31 00:13:30.430006 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:30.429991 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-66pkr" Oct 31 00:13:30.438518 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:30.438487 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-66pkr"] Oct 31 00:13:30.602256 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:30.602229 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1be805c1-5b11-4447-9fcb-264129417960-tls\") pod \"korrel8r-66d6f7f696-66pkr\" (UID: \"1be805c1-5b11-4447-9fcb-264129417960\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-66pkr" Oct 31 00:13:30.602398 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:30.602291 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xxfg\" (UniqueName: \"kubernetes.io/projected/1be805c1-5b11-4447-9fcb-264129417960-kube-api-access-2xxfg\") pod \"korrel8r-66d6f7f696-66pkr\" (UID: \"1be805c1-5b11-4447-9fcb-264129417960\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-66pkr" Oct 31 00:13:30.602398 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:30.602371 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1be805c1-5b11-4447-9fcb-264129417960-config\") pod \"korrel8r-66d6f7f696-66pkr\" (UID: \"1be805c1-5b11-4447-9fcb-264129417960\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-66pkr" Oct 31 00:13:30.702737 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:30.702672 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xxfg\" (UniqueName: \"kubernetes.io/projected/1be805c1-5b11-4447-9fcb-264129417960-kube-api-access-2xxfg\") pod \"korrel8r-66d6f7f696-66pkr\" (UID: \"1be805c1-5b11-4447-9fcb-264129417960\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-66pkr" Oct 31 00:13:30.702737 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:30.702703 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1be805c1-5b11-4447-9fcb-264129417960-config\") pod \"korrel8r-66d6f7f696-66pkr\" (UID: \"1be805c1-5b11-4447-9fcb-264129417960\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-66pkr" Oct 31 00:13:30.702737 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:30.702737 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1be805c1-5b11-4447-9fcb-264129417960-tls\") pod \"korrel8r-66d6f7f696-66pkr\" (UID: \"1be805c1-5b11-4447-9fcb-264129417960\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-66pkr" Oct 31 00:13:30.703515 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:30.703492 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1be805c1-5b11-4447-9fcb-264129417960-config\") pod \"korrel8r-66d6f7f696-66pkr\" (UID: \"1be805c1-5b11-4447-9fcb-264129417960\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-66pkr" Oct 31 00:13:30.705158 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:30.705138 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1be805c1-5b11-4447-9fcb-264129417960-tls\") pod \"korrel8r-66d6f7f696-66pkr\" (UID: \"1be805c1-5b11-4447-9fcb-264129417960\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-66pkr" Oct 31 00:13:30.715727 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:30.715708 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xxfg\" (UniqueName: \"kubernetes.io/projected/1be805c1-5b11-4447-9fcb-264129417960-kube-api-access-2xxfg\") pod \"korrel8r-66d6f7f696-66pkr\" (UID: \"1be805c1-5b11-4447-9fcb-264129417960\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-66pkr" Oct 31 00:13:30.743669 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:30.743650 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-66pkr" Oct 31 00:13:31.022032 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:31.021886 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-66pkr"] Oct 31 00:13:31.024457 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:13:31.024427 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1be805c1_5b11_4447_9fcb_264129417960.slice/crio-f4e7104ffe9487bb2ad7362c4a8a25919885b58ff6c26238e6de03120be3aa90 WatchSource:0}: Error finding container f4e7104ffe9487bb2ad7362c4a8a25919885b58ff6c26238e6de03120be3aa90: Status 404 returned error can't find the container with id f4e7104ffe9487bb2ad7362c4a8a25919885b58ff6c26238e6de03120be3aa90 Oct 31 00:13:31.656641 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:31.656597 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-66pkr" event={"ID":"1be805c1-5b11-4447-9fcb-264129417960","Type":"ContainerStarted","Data":"84babefd75ea786f62d5d8cb6bca66476e80c31b974b9a958ba5437b44739f4f"} Oct 31 00:13:31.657144 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:31.656655 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-66pkr" event={"ID":"1be805c1-5b11-4447-9fcb-264129417960","Type":"ContainerStarted","Data":"f4e7104ffe9487bb2ad7362c4a8a25919885b58ff6c26238e6de03120be3aa90"} Oct 31 00:13:31.657144 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:31.656737 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-66pkr" Oct 31 00:13:31.677423 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:31.677363 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-66pkr" podStartSLOduration=1.677347001 podStartE2EDuration="1.677347001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:31.675627523 +0000 UTC m=+848.312446238" watchObservedRunningTime="2026-10-31 00:13:31.677347001 +0000 UTC m=+848.314165718" Oct 31 00:13:31.944730 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:31.944650 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:13:31.949047 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:31.949027 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:31.966473 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:31.965910 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:13:31.972588 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:31.972567 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\"" Oct 31 00:13:31.972727 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:31.972705 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\"" Oct 31 00:13:31.972727 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:31.972576 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 31 00:13:31.972727 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:31.972597 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\"" Oct 31 00:13:32.033160 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:32.033132 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 31 00:13:32.033285 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:32.033203 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\"" Oct 31 00:13:32.033285 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:32.033217 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\"" Oct 31 00:13:32.033285 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:32.033252 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-9fhd8\"" Oct 31 00:13:32.115623 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:32.115594 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:32.115738 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:32.115633 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:32.115738 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:32.115660 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:32.115807 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:32.115792 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:32.115844 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:32.115818 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:32.115844 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:32.115835 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf8x7\" (UniqueName: \"kubernetes.io/projected/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-kube-api-access-sf8x7\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:32.115949 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:32.115935 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:32.116027 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:32.115994 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:32.217014 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:32.216912 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:32.217014 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:32.216973 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:32.217014 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:32.217010 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:32.217285 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:32.217036 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:32.217285 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:32.217062 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:32.217285 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:32.217087 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf8x7\" (UniqueName: \"kubernetes.io/projected/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-kube-api-access-sf8x7\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:32.217285 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:32.217132 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:32.217285 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:32.217170 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:32.217285 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:32.217256 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway-serving-cert: secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found Oct 31 00:13:32.217586 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:32.217338 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-gateway-serving-cert podName:ce5f8a36-954f-4164-9c7b-7cfff93eee0f nodeName:}" failed. No retries permitted until 2026-10-31 00:13:32.717314364 +0000 UTC m=+849.354133075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-gateway-serving-cert" (UniqueName: "kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-gateway-serving-cert") pod "tempo-data-science-tempomonolithic-0" (UID: "ce5f8a36-954f-4164-9c7b-7cfff93eee0f") : secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found Oct 31 00:13:32.218282 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:32.218249 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:32.218789 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:32.218769 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:32.218872 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:32.218807 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:32.218872 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:32.218823 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:32.220008 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:32.219988 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:32.220127 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:32.220092 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:32.233578 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:32.233555 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf8x7\" (UniqueName: \"kubernetes.io/projected/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-kube-api-access-sf8x7\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:32.722269 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:32.722233 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:32.724575 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:32.724541 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:40.895474 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:40.895439 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:40.902491 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:40.902468 5274 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:40.996767 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:40.996739 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:40.996894 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:40.996839 5274 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:41.000233 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:41.000216 5274 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") DevicePath \"csi-9ce3c80b5d5deac9dfdb953db4fb017080b654a85db76a2937e67cd541d89057\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:41.092129 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:41.092096 5274 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/151619720cb39d40781bf258206fb0be746b77e5f55879e189e4cf0d6e46dfb0/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:41.108696 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:41.108674 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:41.275789 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:41.275720 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:41.556573 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:41.556544 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:13:41.558263 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:13:41.558237 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce5f8a36_954f_4164_9c7b_7cfff93eee0f.slice/crio-ee436f699f50bb1284f9d14d1e3d1694b3e1d71db41a374c05c33ba74c38e0ce WatchSource:0}: Error finding container ee436f699f50bb1284f9d14d1e3d1694b3e1d71db41a374c05c33ba74c38e0ce: Status 404 returned error can't find the container with id ee436f699f50bb1284f9d14d1e3d1694b3e1d71db41a374c05c33ba74c38e0ce Oct 31 00:13:41.711095 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:41.710870 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"ce5f8a36-954f-4164-9c7b-7cfff93eee0f","Type":"ContainerStarted","Data":"f90e1f5cec750c0cc2ca2dd14e07e584170e00a9d2f1ff4b133a1cc223bf7435"} Oct 31 00:13:41.711095 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:41.710933 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"ce5f8a36-954f-4164-9c7b-7cfff93eee0f","Type":"ContainerStarted","Data":"c907c23a2df430d41e6b437a692a51f5c2930b97099b13c30a5b4c4c58c67f67"} Oct 31 00:13:41.711095 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:41.710991 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"ce5f8a36-954f-4164-9c7b-7cfff93eee0f","Type":"ContainerStarted","Data":"ee436f699f50bb1284f9d14d1e3d1694b3e1d71db41a374c05c33ba74c38e0ce"} Oct 31 00:13:42.664381 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:42.664354 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-66pkr" Oct 31 00:13:42.714419 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:42.714392 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-8jrpk"] Oct 31 00:13:42.714601 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:42.714580 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8jrpk" podUID="02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb" containerName="korrel8r" containerID="cri-o://6ef3559a114f12d1f4aacc1175c13b7738775c3f666c9542c526c43fd0f7afbe" gracePeriod=30 Oct 31 00:13:42.722431 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:42.722397 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"ce5f8a36-954f-4164-9c7b-7cfff93eee0f","Type":"ContainerStarted","Data":"ef8d6dccd53bc7cae84866d07b52488889de635c1592e7300d96e65ab6e0e278"} Oct 31 00:13:42.722694 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:42.722672 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:42.722779 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:42.722726 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:42.725233 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:42.725212 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:42.733789 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:42.733771 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:42.762305 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:42.762259 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=11.762246272 podStartE2EDuration="11.762246272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:42.760195464 +0000 UTC m=+859.397014237" watchObservedRunningTime="2026-10-31 00:13:42.762246272 +0000 UTC m=+859.399064987" Oct 31 00:13:42.846242 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:42.846212 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-snfwk"] Oct 31 00:13:42.849690 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:42.849669 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-snfwk" Oct 31 00:13:42.859852 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:42.859824 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-snfwk"] Oct 31 00:13:42.978820 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:42.978796 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8jrpk" Oct 31 00:13:43.015448 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:43.015414 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bf3c0b8e-98ee-4551-9f95-2b4290d189a2-config\") pod \"korrel8r-5cbcc89b8f-snfwk\" (UID: \"bf3c0b8e-98ee-4551-9f95-2b4290d189a2\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-snfwk" Oct 31 00:13:43.015589 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:43.015571 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/bf3c0b8e-98ee-4551-9f95-2b4290d189a2-tls\") pod \"korrel8r-5cbcc89b8f-snfwk\" (UID: \"bf3c0b8e-98ee-4551-9f95-2b4290d189a2\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-snfwk" Oct 31 00:13:43.015642 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:43.015604 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c59xw\" (UniqueName: \"kubernetes.io/projected/bf3c0b8e-98ee-4551-9f95-2b4290d189a2-kube-api-access-c59xw\") pod \"korrel8r-5cbcc89b8f-snfwk\" (UID: \"bf3c0b8e-98ee-4551-9f95-2b4290d189a2\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-snfwk" Oct 31 00:13:43.116371 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:43.116342 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb-tls\" (UniqueName: \"kubernetes.io/secret/02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb-tls\") pod \"02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb\" (UID: \"02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb\") " Oct 31 00:13:43.116510 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:43.116378 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb-kube-api-access-t7b7g\" (UniqueName: \"kubernetes.io/projected/02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb-kube-api-access-t7b7g\") pod \"02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb\" (UID: \"02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb\") " Oct 31 00:13:43.116510 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:43.116489 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb-config\" (UniqueName: \"kubernetes.io/configmap/02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb-config\") pod \"02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb\" (UID: \"02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb\") " Oct 31 00:13:43.116670 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:43.116651 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/bf3c0b8e-98ee-4551-9f95-2b4290d189a2-tls\") pod \"korrel8r-5cbcc89b8f-snfwk\" (UID: \"bf3c0b8e-98ee-4551-9f95-2b4290d189a2\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-snfwk" Oct 31 00:13:43.116728 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:43.116696 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c59xw\" (UniqueName: \"kubernetes.io/projected/bf3c0b8e-98ee-4551-9f95-2b4290d189a2-kube-api-access-c59xw\") pod \"korrel8r-5cbcc89b8f-snfwk\" (UID: \"bf3c0b8e-98ee-4551-9f95-2b4290d189a2\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-snfwk" Oct 31 00:13:43.116782 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:43.116749 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bf3c0b8e-98ee-4551-9f95-2b4290d189a2-config\") pod \"korrel8r-5cbcc89b8f-snfwk\" (UID: \"bf3c0b8e-98ee-4551-9f95-2b4290d189a2\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-snfwk" Oct 31 00:13:43.116852 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:43.116828 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb-config" pod "02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb" (UID: "02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:13:43.117581 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:43.117555 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bf3c0b8e-98ee-4551-9f95-2b4290d189a2-config\") pod \"korrel8r-5cbcc89b8f-snfwk\" (UID: \"bf3c0b8e-98ee-4551-9f95-2b4290d189a2\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-snfwk" Oct 31 00:13:43.118544 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:43.118501 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb-tls" pod "02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb" (UID: "02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:43.118646 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:43.118612 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb-kube-api-access-t7b7g" pod "02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb" (UID: "02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb"). InnerVolumeSpecName "kube-api-access-t7b7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:13:43.119025 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:43.119008 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/bf3c0b8e-98ee-4551-9f95-2b4290d189a2-tls\") pod \"korrel8r-5cbcc89b8f-snfwk\" (UID: \"bf3c0b8e-98ee-4551-9f95-2b4290d189a2\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-snfwk" Oct 31 00:13:43.129114 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:43.129087 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c59xw\" (UniqueName: \"kubernetes.io/projected/bf3c0b8e-98ee-4551-9f95-2b4290d189a2-kube-api-access-c59xw\") pod \"korrel8r-5cbcc89b8f-snfwk\" (UID: \"bf3c0b8e-98ee-4551-9f95-2b4290d189a2\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-snfwk" Oct 31 00:13:43.165767 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:43.165745 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-snfwk" Oct 31 00:13:43.218327 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:43.218296 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:43.218327 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:43.218329 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:43.218532 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:43.218342 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7b7g\" (UniqueName: \"kubernetes.io/projected/02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb-kube-api-access-t7b7g\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:43.440468 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:43.440445 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-snfwk"] Oct 31 00:13:43.442529 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:13:43.442503 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf3c0b8e_98ee_4551_9f95_2b4290d189a2.slice/crio-5119983d524e5c203de251adc6ecd444135e25071e75a3460c033257207bcf78 WatchSource:0}: Error finding container 5119983d524e5c203de251adc6ecd444135e25071e75a3460c033257207bcf78: Status 404 returned error can't find the container with id 5119983d524e5c203de251adc6ecd444135e25071e75a3460c033257207bcf78 Oct 31 00:13:43.536279 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:43.536242 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:13:43.728321 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:43.728230 5274 generic.go:356] "Generic (PLEG): container finished" podID="02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb" containerID="6ef3559a114f12d1f4aacc1175c13b7738775c3f666c9542c526c43fd0f7afbe" exitCode=0 Oct 31 00:13:43.728321 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:43.728274 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8jrpk" Oct 31 00:13:43.728321 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:43.728312 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8jrpk" event={"ID":"02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb","Type":"ContainerDied","Data":"6ef3559a114f12d1f4aacc1175c13b7738775c3f666c9542c526c43fd0f7afbe"} Oct 31 00:13:43.728831 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:43.728368 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8jrpk" event={"ID":"02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb","Type":"ContainerDied","Data":"3e64fbd14c3864ba692a9cfc52a2b20c37b221c5261e85d7a86b14c56b631146"} Oct 31 00:13:43.728831 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:43.728397 5274 scope.go:122] "RemoveContainer" containerID="6ef3559a114f12d1f4aacc1175c13b7738775c3f666c9542c526c43fd0f7afbe" Oct 31 00:13:43.730225 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:43.730191 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-snfwk" event={"ID":"bf3c0b8e-98ee-4551-9f95-2b4290d189a2","Type":"ContainerStarted","Data":"39d4661ab9dd84d3d4855f81237ac32c6993792ddc89705fa22054be2ff29b63"} Oct 31 00:13:43.730357 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:43.730276 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-snfwk" event={"ID":"bf3c0b8e-98ee-4551-9f95-2b4290d189a2","Type":"ContainerStarted","Data":"5119983d524e5c203de251adc6ecd444135e25071e75a3460c033257207bcf78"} Oct 31 00:13:43.730426 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:43.730372 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:43.730426 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:43.730407 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-snfwk" Oct 31 00:13:43.730555 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:43.730534 5274 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" secret="" err="secret \"tempo-data-science-tempomonolithic-dockercfg-9fhd8\" not found" Oct 31 00:13:43.737364 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:43.737349 5274 scope.go:122] "RemoveContainer" containerID="6ef3559a114f12d1f4aacc1175c13b7738775c3f666c9542c526c43fd0f7afbe" Oct 31 00:13:43.737635 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:43.737616 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ef3559a114f12d1f4aacc1175c13b7738775c3f666c9542c526c43fd0f7afbe\": container with ID starting with 6ef3559a114f12d1f4aacc1175c13b7738775c3f666c9542c526c43fd0f7afbe not found: ID does not exist" containerID="6ef3559a114f12d1f4aacc1175c13b7738775c3f666c9542c526c43fd0f7afbe" Oct 31 00:13:43.737703 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:43.737646 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ef3559a114f12d1f4aacc1175c13b7738775c3f666c9542c526c43fd0f7afbe"} err="failed to get container status \"6ef3559a114f12d1f4aacc1175c13b7738775c3f666c9542c526c43fd0f7afbe\": rpc error: code = NotFound desc = could not find container \"6ef3559a114f12d1f4aacc1175c13b7738775c3f666c9542c526c43fd0f7afbe\": container with ID starting with 6ef3559a114f12d1f4aacc1175c13b7738775c3f666c9542c526c43fd0f7afbe not found: ID does not exist" Oct 31 00:13:43.745854 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:43.745833 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-8jrpk"] Oct 31 00:13:43.750562 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:43.750543 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-8jrpk"] Oct 31 00:13:43.767124 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:43.767085 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-snfwk" podStartSLOduration=1.7670721409999999 podStartE2EDuration="1.767072141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:43.76580583 +0000 UTC m=+860.402624546" watchObservedRunningTime="2026-10-31 00:13:43.767072141 +0000 UTC m=+860.403890892" Oct 31 00:13:43.823104 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:43.823066 5274 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: configmap "tempo-data-science-tempomonolithic-gateway" not found Oct 31 00:13:43.823104 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:43.823093 5274 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-ca-bundle: configmap "tempo-data-science-tempomonolithic-ca-bundle" not found Oct 31 00:13:43.823104 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:43.823093 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-mtls: secret "tempo-data-science-tempomonolithic-mtls" not found Oct 31 00:13:43.823342 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:43.823141 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-gateway-rbac podName:ce5f8a36-954f-4164-9c7b-7cfff93eee0f nodeName:}" failed. No retries permitted until 2026-10-31 00:13:44.323119616 +0000 UTC m=+860.959938332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "gateway-rbac" (UniqueName: "kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-gateway-rbac") pod "tempo-data-science-tempomonolithic-0" (UID: "ce5f8a36-954f-4164-9c7b-7cfff93eee0f") : configmap "tempo-data-science-tempomonolithic-gateway" not found Oct 31 00:13:43.823342 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:43.823075 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway-serving-cert: secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found Oct 31 00:13:43.823342 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:43.823167 5274 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-config: configmap "tempo-data-science-tempomonolithic-config" not found Oct 31 00:13:43.823342 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:43.823182 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: secret "tempo-data-science-tempomonolithic-gateway" not found Oct 31 00:13:43.823342 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:43.823190 5274 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-serving-cabundle: configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found Oct 31 00:13:43.823342 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:43.823205 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-ca-bundle podName:ce5f8a36-954f-4164-9c7b-7cfff93eee0f nodeName:}" failed. No retries permitted until 2026-10-31 00:13:44.323193523 +0000 UTC m=+860.960012232 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-ca-bundle" (UniqueName: "kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-ca-bundle") pod "tempo-data-science-tempomonolithic-0" (UID: "ce5f8a36-954f-4164-9c7b-7cfff93eee0f") : configmap "tempo-data-science-tempomonolithic-ca-bundle" not found Oct 31 00:13:43.823342 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:43.823225 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-mtls podName:ce5f8a36-954f-4164-9c7b-7cfff93eee0f nodeName:}" failed. No retries permitted until 2026-10-31 00:13:44.323215974 +0000 UTC m=+860.960034667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-mtls" (UniqueName: "kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-mtls") pod "tempo-data-science-tempomonolithic-0" (UID: "ce5f8a36-954f-4164-9c7b-7cfff93eee0f") : secret "tempo-data-science-tempomonolithic-mtls" not found Oct 31 00:13:43.823342 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:43.823243 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-serving-cabundle podName:ce5f8a36-954f-4164-9c7b-7cfff93eee0f nodeName:}" failed. No retries permitted until 2026-10-31 00:13:44.323234431 +0000 UTC m=+860.960053124 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-serving-cabundle" (UniqueName: "kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-serving-cabundle") pod "tempo-data-science-tempomonolithic-0" (UID: "ce5f8a36-954f-4164-9c7b-7cfff93eee0f") : configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found Oct 31 00:13:43.823342 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:43.823260 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-gateway-serving-cert podName:ce5f8a36-954f-4164-9c7b-7cfff93eee0f nodeName:}" failed. No retries permitted until 2026-10-31 00:13:44.3232518 +0000 UTC m=+860.960070494 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-gateway-serving-cert" (UniqueName: "kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-gateway-serving-cert") pod "tempo-data-science-tempomonolithic-0" (UID: "ce5f8a36-954f-4164-9c7b-7cfff93eee0f") : secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found Oct 31 00:13:43.823342 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:43.823296 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-conf podName:ce5f8a36-954f-4164-9c7b-7cfff93eee0f nodeName:}" failed. No retries permitted until 2026-10-31 00:13:44.323275033 +0000 UTC m=+860.960093727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-conf" (UniqueName: "kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-conf") pod "tempo-data-science-tempomonolithic-0" (UID: "ce5f8a36-954f-4164-9c7b-7cfff93eee0f") : configmap "tempo-data-science-tempomonolithic-config" not found Oct 31 00:13:43.823342 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:43.823311 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-gateway-tenants podName:ce5f8a36-954f-4164-9c7b-7cfff93eee0f nodeName:}" failed. No retries permitted until 2026-10-31 00:13:44.32330541 +0000 UTC m=+860.960124104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "gateway-tenants" (UniqueName: "kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-gateway-tenants") pod "tempo-data-science-tempomonolithic-0" (UID: "ce5f8a36-954f-4164-9c7b-7cfff93eee0f") : secret "tempo-data-science-tempomonolithic-gateway" not found Oct 31 00:13:44.326635 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:44.326605 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway-serving-cert: secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found Oct 31 00:13:44.326801 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:44.326665 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-gateway-serving-cert podName:ce5f8a36-954f-4164-9c7b-7cfff93eee0f nodeName:}" failed. No retries permitted until 2026-10-31 00:13:45.326651642 +0000 UTC m=+861.963470360 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-gateway-serving-cert" (UniqueName: "kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-gateway-serving-cert") pod "tempo-data-science-tempomonolithic-0" (UID: "ce5f8a36-954f-4164-9c7b-7cfff93eee0f") : secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found Oct 31 00:13:44.326801 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:44.326672 5274 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-ca-bundle: configmap "tempo-data-science-tempomonolithic-ca-bundle" not found Oct 31 00:13:44.326801 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:44.326606 5274 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-config: configmap "tempo-data-science-tempomonolithic-config" not found Oct 31 00:13:44.326801 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:44.326609 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: secret "tempo-data-science-tempomonolithic-gateway" not found Oct 31 00:13:44.326801 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:44.326717 5274 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: configmap "tempo-data-science-tempomonolithic-gateway" not found Oct 31 00:13:44.326801 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:44.326728 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-ca-bundle podName:ce5f8a36-954f-4164-9c7b-7cfff93eee0f nodeName:}" failed. No retries permitted until 2026-10-31 00:13:45.32671216 +0000 UTC m=+861.963530889 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-ca-bundle" (UniqueName: "kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-ca-bundle") pod "tempo-data-science-tempomonolithic-0" (UID: "ce5f8a36-954f-4164-9c7b-7cfff93eee0f") : configmap "tempo-data-science-tempomonolithic-ca-bundle" not found Oct 31 00:13:44.326801 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:44.326745 5274 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-serving-cabundle: configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found Oct 31 00:13:44.326801 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:44.326751 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-gateway-tenants podName:ce5f8a36-954f-4164-9c7b-7cfff93eee0f nodeName:}" failed. No retries permitted until 2026-10-31 00:13:45.326738156 +0000 UTC m=+861.963556851 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "gateway-tenants" (UniqueName: "kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-gateway-tenants") pod "tempo-data-science-tempomonolithic-0" (UID: "ce5f8a36-954f-4164-9c7b-7cfff93eee0f") : secret "tempo-data-science-tempomonolithic-gateway" not found Oct 31 00:13:44.326801 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:44.326767 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-conf podName:ce5f8a36-954f-4164-9c7b-7cfff93eee0f nodeName:}" failed. No retries permitted until 2026-10-31 00:13:45.326758348 +0000 UTC m=+861.963577046 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-conf" (UniqueName: "kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-conf") pod "tempo-data-science-tempomonolithic-0" (UID: "ce5f8a36-954f-4164-9c7b-7cfff93eee0f") : configmap "tempo-data-science-tempomonolithic-config" not found Oct 31 00:13:44.326801 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:44.326784 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-serving-cabundle podName:ce5f8a36-954f-4164-9c7b-7cfff93eee0f nodeName:}" failed. No retries permitted until 2026-10-31 00:13:45.32677258 +0000 UTC m=+861.963591302 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-serving-cabundle" (UniqueName: "kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-serving-cabundle") pod "tempo-data-science-tempomonolithic-0" (UID: "ce5f8a36-954f-4164-9c7b-7cfff93eee0f") : configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found Oct 31 00:13:44.326801 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:44.326787 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-mtls: secret "tempo-data-science-tempomonolithic-mtls" not found Oct 31 00:13:44.326801 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:44.326799 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-gateway-rbac podName:ce5f8a36-954f-4164-9c7b-7cfff93eee0f nodeName:}" failed. No retries permitted until 2026-10-31 00:13:45.326791522 +0000 UTC m=+861.963610216 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "gateway-rbac" (UniqueName: "kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-gateway-rbac") pod "tempo-data-science-tempomonolithic-0" (UID: "ce5f8a36-954f-4164-9c7b-7cfff93eee0f") : configmap "tempo-data-science-tempomonolithic-gateway" not found Oct 31 00:13:44.326801 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:44.326812 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-mtls podName:ce5f8a36-954f-4164-9c7b-7cfff93eee0f nodeName:}" failed. No retries permitted until 2026-10-31 00:13:45.326804903 +0000 UTC m=+861.963623601 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-mtls" (UniqueName: "kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-mtls") pod "tempo-data-science-tempomonolithic-0" (UID: "ce5f8a36-954f-4164-9c7b-7cfff93eee0f") : secret "tempo-data-science-tempomonolithic-mtls" not found Oct 31 00:13:44.736541 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:44.736498 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="ce5f8a36-954f-4164-9c7b-7cfff93eee0f" containerName="tempo" containerID="cri-o://c907c23a2df430d41e6b437a692a51f5c2930b97099b13c30a5b4c4c58c67f67" gracePeriod=30 Oct 31 00:13:44.736541 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:44.736541 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="ce5f8a36-954f-4164-9c7b-7cfff93eee0f" containerName="tempo-gateway-opa" containerID="cri-o://ef8d6dccd53bc7cae84866d07b52488889de635c1592e7300d96e65ab6e0e278" gracePeriod=30 Oct 31 00:13:44.737111 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:44.736572 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="ce5f8a36-954f-4164-9c7b-7cfff93eee0f" containerName="tempo-gateway" containerID="cri-o://f90e1f5cec750c0cc2ca2dd14e07e584170e00a9d2f1ff4b133a1cc223bf7435" gracePeriod=30 Oct 31 00:13:45.336467 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:45.336435 5274 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-serving-cabundle: configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found Oct 31 00:13:45.336467 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:45.336450 5274 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-config: configmap "tempo-data-science-tempomonolithic-config" not found Oct 31 00:13:45.336720 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:45.336491 5274 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-ca-bundle: configmap "tempo-data-science-tempomonolithic-ca-bundle" not found Oct 31 00:13:45.336720 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:45.336504 5274 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: configmap "tempo-data-science-tempomonolithic-gateway" not found Oct 31 00:13:45.336720 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:45.336491 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: secret "tempo-data-science-tempomonolithic-gateway" not found Oct 31 00:13:45.336720 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:45.336542 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-mtls: secret "tempo-data-science-tempomonolithic-mtls" not found Oct 31 00:13:45.336720 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:45.336508 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway-serving-cert: secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found Oct 31 00:13:45.336720 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:45.336513 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-serving-cabundle podName:ce5f8a36-954f-4164-9c7b-7cfff93eee0f nodeName:}" failed. No retries permitted until 2026-10-31 00:13:47.336493536 +0000 UTC m=+863.973312255 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-serving-cabundle" (UniqueName: "kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-serving-cabundle") pod "tempo-data-science-tempomonolithic-0" (UID: "ce5f8a36-954f-4164-9c7b-7cfff93eee0f") : configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found Oct 31 00:13:45.336720 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:45.336614 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-conf podName:ce5f8a36-954f-4164-9c7b-7cfff93eee0f nodeName:}" failed. No retries permitted until 2026-10-31 00:13:47.336602306 +0000 UTC m=+863.973421003 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-conf" (UniqueName: "kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-conf") pod "tempo-data-science-tempomonolithic-0" (UID: "ce5f8a36-954f-4164-9c7b-7cfff93eee0f") : configmap "tempo-data-science-tempomonolithic-config" not found Oct 31 00:13:45.336720 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:45.336626 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-ca-bundle podName:ce5f8a36-954f-4164-9c7b-7cfff93eee0f nodeName:}" failed. No retries permitted until 2026-10-31 00:13:47.336619679 +0000 UTC m=+863.973438376 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-ca-bundle" (UniqueName: "kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-ca-bundle") pod "tempo-data-science-tempomonolithic-0" (UID: "ce5f8a36-954f-4164-9c7b-7cfff93eee0f") : configmap "tempo-data-science-tempomonolithic-ca-bundle" not found Oct 31 00:13:45.336720 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:45.336643 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-gateway-rbac podName:ce5f8a36-954f-4164-9c7b-7cfff93eee0f nodeName:}" failed. No retries permitted until 2026-10-31 00:13:47.33663063 +0000 UTC m=+863.973449324 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "gateway-rbac" (UniqueName: "kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-gateway-rbac") pod "tempo-data-science-tempomonolithic-0" (UID: "ce5f8a36-954f-4164-9c7b-7cfff93eee0f") : configmap "tempo-data-science-tempomonolithic-gateway" not found Oct 31 00:13:45.336720 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:45.336656 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-gateway-tenants podName:ce5f8a36-954f-4164-9c7b-7cfff93eee0f nodeName:}" failed. No retries permitted until 2026-10-31 00:13:47.336650144 +0000 UTC m=+863.973468838 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "gateway-tenants" (UniqueName: "kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-gateway-tenants") pod "tempo-data-science-tempomonolithic-0" (UID: "ce5f8a36-954f-4164-9c7b-7cfff93eee0f") : secret "tempo-data-science-tempomonolithic-gateway" not found Oct 31 00:13:45.336720 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:45.336670 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-mtls podName:ce5f8a36-954f-4164-9c7b-7cfff93eee0f nodeName:}" failed. No retries permitted until 2026-10-31 00:13:47.33666483 +0000 UTC m=+863.973483524 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-mtls" (UniqueName: "kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-mtls") pod "tempo-data-science-tempomonolithic-0" (UID: "ce5f8a36-954f-4164-9c7b-7cfff93eee0f") : secret "tempo-data-science-tempomonolithic-mtls" not found Oct 31 00:13:45.336720 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:45.336682 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-gateway-serving-cert podName:ce5f8a36-954f-4164-9c7b-7cfff93eee0f nodeName:}" failed. No retries permitted until 2026-10-31 00:13:47.33667727 +0000 UTC m=+863.973495969 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-gateway-serving-cert" (UniqueName: "kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-gateway-serving-cert") pod "tempo-data-science-tempomonolithic-0" (UID: "ce5f8a36-954f-4164-9c7b-7cfff93eee0f") : secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found Oct 31 00:13:45.483514 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:45.483481 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb" path="/var/lib/kubelet/pods/02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb/volumes" Oct 31 00:13:45.744394 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:45.744363 5274 generic.go:356] "Generic (PLEG): container finished" podID="ce5f8a36-954f-4164-9c7b-7cfff93eee0f" containerID="ef8d6dccd53bc7cae84866d07b52488889de635c1592e7300d96e65ab6e0e278" exitCode=0 Oct 31 00:13:45.744394 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:45.744387 5274 generic.go:356] "Generic (PLEG): container finished" podID="ce5f8a36-954f-4164-9c7b-7cfff93eee0f" containerID="c907c23a2df430d41e6b437a692a51f5c2930b97099b13c30a5b4c4c58c67f67" exitCode=0 Oct 31 00:13:45.744798 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:45.744438 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"ce5f8a36-954f-4164-9c7b-7cfff93eee0f","Type":"ContainerDied","Data":"ef8d6dccd53bc7cae84866d07b52488889de635c1592e7300d96e65ab6e0e278"} Oct 31 00:13:45.744798 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:45.744487 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"ce5f8a36-954f-4164-9c7b-7cfff93eee0f","Type":"ContainerDied","Data":"c907c23a2df430d41e6b437a692a51f5c2930b97099b13c30a5b4c4c58c67f67"} Oct 31 00:13:45.987321 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:45.987297 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:46.143240 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.143213 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") " Oct 31 00:13:46.143415 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.143248 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-ca-bundle\") pod \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") " Oct 31 00:13:46.143415 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.143280 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-conf\") pod \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") " Oct 31 00:13:46.143415 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.143299 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-gateway-tenants\") pod \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") " Oct 31 00:13:46.143415 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.143318 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-gateway-rbac\") pod \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") " Oct 31 00:13:46.143415 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.143361 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-kube-api-access-sf8x7\" (UniqueName: \"kubernetes.io/projected/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-kube-api-access-sf8x7\") pod \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") " Oct 31 00:13:46.143691 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.143425 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-mtls\") pod \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") " Oct 31 00:13:46.143691 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.143513 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") " Oct 31 00:13:46.143691 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.143595 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\" (UID: \"ce5f8a36-954f-4164-9c7b-7cfff93eee0f\") " Oct 31 00:13:46.143691 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.143619 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-serving-cabundle" pod "ce5f8a36-954f-4164-9c7b-7cfff93eee0f" (UID: "ce5f8a36-954f-4164-9c7b-7cfff93eee0f"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:13:46.143691 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.143681 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-conf" pod "ce5f8a36-954f-4164-9c7b-7cfff93eee0f" (UID: "ce5f8a36-954f-4164-9c7b-7cfff93eee0f"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:13:46.143943 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.143755 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-gateway-rbac" pod "ce5f8a36-954f-4164-9c7b-7cfff93eee0f" (UID: "ce5f8a36-954f-4164-9c7b-7cfff93eee0f"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:13:46.143943 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.143882 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:46.143943 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.143899 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-conf\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:46.143943 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.143912 5274 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-gateway-rbac\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:46.144201 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.143998 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-ca-bundle" pod "ce5f8a36-954f-4164-9c7b-7cfff93eee0f" (UID: "ce5f8a36-954f-4164-9c7b-7cfff93eee0f"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:13:46.145786 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.145737 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "ce5f8a36-954f-4164-9c7b-7cfff93eee0f" (UID: "ce5f8a36-954f-4164-9c7b-7cfff93eee0f"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:46.145905 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.145830 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-mtls" pod "ce5f8a36-954f-4164-9c7b-7cfff93eee0f" (UID: "ce5f8a36-954f-4164-9c7b-7cfff93eee0f"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:46.145905 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.145839 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-kube-api-access-sf8x7" pod "ce5f8a36-954f-4164-9c7b-7cfff93eee0f" (UID: "ce5f8a36-954f-4164-9c7b-7cfff93eee0f"). InnerVolumeSpecName "kube-api-access-sf8x7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:13:46.145905 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.145895 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-gateway-tenants" pod "ce5f8a36-954f-4164-9c7b-7cfff93eee0f" (UID: "ce5f8a36-954f-4164-9c7b-7cfff93eee0f"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:46.146750 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.146732 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062" pod "ce5f8a36-954f-4164-9c7b-7cfff93eee0f" (UID: "ce5f8a36-954f-4164-9c7b-7cfff93eee0f"). InnerVolumeSpecName "pvc-4491ac70-4597-43a7-a792-f0813314e433". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:13:46.245236 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.245198 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:46.245236 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.245235 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:46.245417 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.245272 5274 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" " Oct 31 00:13:46.245417 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.245288 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:46.245417 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.245302 5274 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-gateway-tenants\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:46.245417 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.245316 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf8x7\" (UniqueName: \"kubernetes.io/projected/ce5f8a36-954f-4164-9c7b-7cfff93eee0f-kube-api-access-sf8x7\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:46.256676 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.256660 5274 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-4491ac70-4597-43a7-a792-f0813314e433" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062") on node "ip-10-0-129-70.us-east-2.compute.internal" Oct 31 00:13:46.346262 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.346230 5274 reconciler_common.go:299] "Volume detached for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"csi-9ce3c80b5d5deac9dfdb953db4fb017080b654a85db76a2937e67cd541d89057\"" Oct 31 00:13:46.751758 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.751726 5274 generic.go:356] "Generic (PLEG): container finished" podID="ce5f8a36-954f-4164-9c7b-7cfff93eee0f" containerID="f90e1f5cec750c0cc2ca2dd14e07e584170e00a9d2f1ff4b133a1cc223bf7435" exitCode=0 Oct 31 00:13:46.752275 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.751790 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:13:46.752275 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.751814 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"ce5f8a36-954f-4164-9c7b-7cfff93eee0f","Type":"ContainerDied","Data":"f90e1f5cec750c0cc2ca2dd14e07e584170e00a9d2f1ff4b133a1cc223bf7435"} Oct 31 00:13:46.752275 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.751874 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"ce5f8a36-954f-4164-9c7b-7cfff93eee0f","Type":"ContainerDied","Data":"ee436f699f50bb1284f9d14d1e3d1694b3e1d71db41a374c05c33ba74c38e0ce"} Oct 31 00:13:46.752275 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.751898 5274 scope.go:122] "RemoveContainer" containerID="ef8d6dccd53bc7cae84866d07b52488889de635c1592e7300d96e65ab6e0e278" Oct 31 00:13:46.760932 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.760915 5274 scope.go:122] "RemoveContainer" containerID="f90e1f5cec750c0cc2ca2dd14e07e584170e00a9d2f1ff4b133a1cc223bf7435" Oct 31 00:13:46.769018 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.769000 5274 scope.go:122] "RemoveContainer" containerID="c907c23a2df430d41e6b437a692a51f5c2930b97099b13c30a5b4c4c58c67f67" Oct 31 00:13:46.776680 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.776664 5274 scope.go:122] "RemoveContainer" containerID="ef8d6dccd53bc7cae84866d07b52488889de635c1592e7300d96e65ab6e0e278" Oct 31 00:13:46.776936 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:46.776920 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef8d6dccd53bc7cae84866d07b52488889de635c1592e7300d96e65ab6e0e278\": container with ID starting with ef8d6dccd53bc7cae84866d07b52488889de635c1592e7300d96e65ab6e0e278 not found: ID does not exist" containerID="ef8d6dccd53bc7cae84866d07b52488889de635c1592e7300d96e65ab6e0e278" Oct 31 00:13:46.777013 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.776943 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef8d6dccd53bc7cae84866d07b52488889de635c1592e7300d96e65ab6e0e278"} err="failed to get container status \"ef8d6dccd53bc7cae84866d07b52488889de635c1592e7300d96e65ab6e0e278\": rpc error: code = NotFound desc = could not find container \"ef8d6dccd53bc7cae84866d07b52488889de635c1592e7300d96e65ab6e0e278\": container with ID starting with ef8d6dccd53bc7cae84866d07b52488889de635c1592e7300d96e65ab6e0e278 not found: ID does not exist" Oct 31 00:13:46.777013 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.776972 5274 scope.go:122] "RemoveContainer" containerID="f90e1f5cec750c0cc2ca2dd14e07e584170e00a9d2f1ff4b133a1cc223bf7435" Oct 31 00:13:46.777246 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:46.777216 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f90e1f5cec750c0cc2ca2dd14e07e584170e00a9d2f1ff4b133a1cc223bf7435\": container with ID starting with f90e1f5cec750c0cc2ca2dd14e07e584170e00a9d2f1ff4b133a1cc223bf7435 not found: ID does not exist" containerID="f90e1f5cec750c0cc2ca2dd14e07e584170e00a9d2f1ff4b133a1cc223bf7435" Oct 31 00:13:46.777357 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.777249 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f90e1f5cec750c0cc2ca2dd14e07e584170e00a9d2f1ff4b133a1cc223bf7435"} err="failed to get container status \"f90e1f5cec750c0cc2ca2dd14e07e584170e00a9d2f1ff4b133a1cc223bf7435\": rpc error: code = NotFound desc = could not find container \"f90e1f5cec750c0cc2ca2dd14e07e584170e00a9d2f1ff4b133a1cc223bf7435\": container with ID starting with f90e1f5cec750c0cc2ca2dd14e07e584170e00a9d2f1ff4b133a1cc223bf7435 not found: ID does not exist" Oct 31 00:13:46.777357 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.777270 5274 scope.go:122] "RemoveContainer" containerID="c907c23a2df430d41e6b437a692a51f5c2930b97099b13c30a5b4c4c58c67f67" Oct 31 00:13:46.777798 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:46.777769 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c907c23a2df430d41e6b437a692a51f5c2930b97099b13c30a5b4c4c58c67f67\": container with ID starting with c907c23a2df430d41e6b437a692a51f5c2930b97099b13c30a5b4c4c58c67f67 not found: ID does not exist" containerID="c907c23a2df430d41e6b437a692a51f5c2930b97099b13c30a5b4c4c58c67f67" Oct 31 00:13:46.777871 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.777796 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c907c23a2df430d41e6b437a692a51f5c2930b97099b13c30a5b4c4c58c67f67"} err="failed to get container status \"c907c23a2df430d41e6b437a692a51f5c2930b97099b13c30a5b4c4c58c67f67\": rpc error: code = NotFound desc = could not find container \"c907c23a2df430d41e6b437a692a51f5c2930b97099b13c30a5b4c4c58c67f67\": container with ID starting with c907c23a2df430d41e6b437a692a51f5c2930b97099b13c30a5b4c4c58c67f67 not found: ID does not exist" Oct 31 00:13:46.780056 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.780033 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:13:46.783818 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:46.783797 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:13:47.481499 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:47.481468 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce5f8a36-954f-4164-9c7b-7cfff93eee0f" path="/var/lib/kubelet/pods/ce5f8a36-954f-4164-9c7b-7cfff93eee0f/volumes" Oct 31 00:13:50.900015 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:50.899982 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:13:54.738827 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:54.738796 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-snfwk" Oct 31 00:13:54.787382 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:54.787357 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-66pkr"] Oct 31 00:13:54.787607 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:54.787545 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-66pkr" podUID="1be805c1-5b11-4447-9fcb-264129417960" containerName="korrel8r" containerID="cri-o://84babefd75ea786f62d5d8cb6bca66476e80c31b974b9a958ba5437b44739f4f" gracePeriod=30 Oct 31 00:13:55.034595 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:55.034572 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-66pkr" Oct 31 00:13:55.122010 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:55.121977 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1be805c1-5b11-4447-9fcb-264129417960-tls\" (UniqueName: \"kubernetes.io/secret/1be805c1-5b11-4447-9fcb-264129417960-tls\") pod \"1be805c1-5b11-4447-9fcb-264129417960\" (UID: \"1be805c1-5b11-4447-9fcb-264129417960\") " Oct 31 00:13:55.122170 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:55.122024 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1be805c1-5b11-4447-9fcb-264129417960-kube-api-access-2xxfg\" (UniqueName: \"kubernetes.io/projected/1be805c1-5b11-4447-9fcb-264129417960-kube-api-access-2xxfg\") pod \"1be805c1-5b11-4447-9fcb-264129417960\" (UID: \"1be805c1-5b11-4447-9fcb-264129417960\") " Oct 31 00:13:55.122170 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:55.122058 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1be805c1-5b11-4447-9fcb-264129417960-config\" (UniqueName: \"kubernetes.io/configmap/1be805c1-5b11-4447-9fcb-264129417960-config\") pod \"1be805c1-5b11-4447-9fcb-264129417960\" (UID: \"1be805c1-5b11-4447-9fcb-264129417960\") " Oct 31 00:13:55.122509 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:55.122483 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1be805c1-5b11-4447-9fcb-264129417960-config" pod "1be805c1-5b11-4447-9fcb-264129417960" (UID: "1be805c1-5b11-4447-9fcb-264129417960"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:13:55.124132 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:55.124105 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1be805c1-5b11-4447-9fcb-264129417960-tls" pod "1be805c1-5b11-4447-9fcb-264129417960" (UID: "1be805c1-5b11-4447-9fcb-264129417960"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:55.124255 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:55.124193 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1be805c1-5b11-4447-9fcb-264129417960-kube-api-access-2xxfg" pod "1be805c1-5b11-4447-9fcb-264129417960" (UID: "1be805c1-5b11-4447-9fcb-264129417960"). InnerVolumeSpecName "kube-api-access-2xxfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:13:55.222589 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:55.222555 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xxfg\" (UniqueName: \"kubernetes.io/projected/1be805c1-5b11-4447-9fcb-264129417960-kube-api-access-2xxfg\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:55.222589 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:55.222585 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1be805c1-5b11-4447-9fcb-264129417960-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:55.222734 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:55.222598 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1be805c1-5b11-4447-9fcb-264129417960-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:55.800928 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:55.800833 5274 generic.go:356] "Generic (PLEG): container finished" podID="1be805c1-5b11-4447-9fcb-264129417960" containerID="84babefd75ea786f62d5d8cb6bca66476e80c31b974b9a958ba5437b44739f4f" exitCode=0 Oct 31 00:13:55.800928 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:55.800876 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-66pkr" event={"ID":"1be805c1-5b11-4447-9fcb-264129417960","Type":"ContainerDied","Data":"84babefd75ea786f62d5d8cb6bca66476e80c31b974b9a958ba5437b44739f4f"} Oct 31 00:13:55.800928 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:55.800905 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-66pkr" Oct 31 00:13:55.800928 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:55.800915 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-66pkr" event={"ID":"1be805c1-5b11-4447-9fcb-264129417960","Type":"ContainerDied","Data":"f4e7104ffe9487bb2ad7362c4a8a25919885b58ff6c26238e6de03120be3aa90"} Oct 31 00:13:55.801523 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:55.800937 5274 scope.go:122] "RemoveContainer" containerID="84babefd75ea786f62d5d8cb6bca66476e80c31b974b9a958ba5437b44739f4f" Oct 31 00:13:55.809319 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:55.809304 5274 scope.go:122] "RemoveContainer" containerID="84babefd75ea786f62d5d8cb6bca66476e80c31b974b9a958ba5437b44739f4f" Oct 31 00:13:55.809571 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:13:55.809553 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84babefd75ea786f62d5d8cb6bca66476e80c31b974b9a958ba5437b44739f4f\": container with ID starting with 84babefd75ea786f62d5d8cb6bca66476e80c31b974b9a958ba5437b44739f4f not found: ID does not exist" containerID="84babefd75ea786f62d5d8cb6bca66476e80c31b974b9a958ba5437b44739f4f" Oct 31 00:13:55.809617 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:55.809578 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84babefd75ea786f62d5d8cb6bca66476e80c31b974b9a958ba5437b44739f4f"} err="failed to get container status \"84babefd75ea786f62d5d8cb6bca66476e80c31b974b9a958ba5437b44739f4f\": rpc error: code = NotFound desc = could not find container \"84babefd75ea786f62d5d8cb6bca66476e80c31b974b9a958ba5437b44739f4f\": container with ID starting with 84babefd75ea786f62d5d8cb6bca66476e80c31b974b9a958ba5437b44739f4f not found: ID does not exist" Oct 31 00:13:55.818257 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:55.818234 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-66pkr"] Oct 31 00:13:55.822770 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:55.822752 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-66pkr"] Oct 31 00:13:57.481141 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:13:57.481105 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1be805c1-5b11-4447-9fcb-264129417960" path="/var/lib/kubelet/pods/1be805c1-5b11-4447-9fcb-264129417960/volumes" Oct 31 00:14:19.870365 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:19.870329 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-2"] Oct 31 00:14:19.870842 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:19.870721 5274 cpu_manager.go:414] "containerMap: removing container" podUID="02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb" containerName="korrel8r" Oct 31 00:14:19.870842 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:19.870738 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb" containerName="korrel8r" Oct 31 00:14:19.870842 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:19.870759 5274 cpu_manager.go:414] "containerMap: removing container" podUID="ce5f8a36-954f-4164-9c7b-7cfff93eee0f" containerName="tempo-gateway" Oct 31 00:14:19.870842 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:19.870768 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce5f8a36-954f-4164-9c7b-7cfff93eee0f" containerName="tempo-gateway" Oct 31 00:14:19.870842 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:19.870781 5274 cpu_manager.go:414] "containerMap: removing container" podUID="ce5f8a36-954f-4164-9c7b-7cfff93eee0f" containerName="tempo-gateway-opa" Oct 31 00:14:19.871405 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:19.870789 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce5f8a36-954f-4164-9c7b-7cfff93eee0f" containerName="tempo-gateway-opa" Oct 31 00:14:19.871527 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:19.871433 5274 cpu_manager.go:414] "containerMap: removing container" podUID="1be805c1-5b11-4447-9fcb-264129417960" containerName="korrel8r" Oct 31 00:14:19.871527 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:19.871446 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1be805c1-5b11-4447-9fcb-264129417960" containerName="korrel8r" Oct 31 00:14:19.871527 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:19.871457 5274 cpu_manager.go:414] "containerMap: removing container" podUID="ce5f8a36-954f-4164-9c7b-7cfff93eee0f" containerName="tempo" Oct 31 00:14:19.871527 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:19.871463 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce5f8a36-954f-4164-9c7b-7cfff93eee0f" containerName="tempo" Oct 31 00:14:19.871688 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:19.871613 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="02bd33f0-a57a-4c9f-bc4d-7e2e8e5ebafb" containerName="korrel8r" Oct 31 00:14:19.871688 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:19.871624 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce5f8a36-954f-4164-9c7b-7cfff93eee0f" containerName="tempo-gateway" Oct 31 00:14:19.871688 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:19.871630 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce5f8a36-954f-4164-9c7b-7cfff93eee0f" containerName="tempo-gateway-opa" Oct 31 00:14:19.871688 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:19.871642 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="1be805c1-5b11-4447-9fcb-264129417960" containerName="korrel8r" Oct 31 00:14:19.871688 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:19.871650 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce5f8a36-954f-4164-9c7b-7cfff93eee0f" containerName="tempo" Oct 31 00:14:19.875027 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:19.875009 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 31 00:14:19.883115 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:19.883096 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:14:19.883216 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:19.883154 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:14:19.883257 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:19.883228 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:14:19.883299 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:19.883273 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-g8x7h\"" Oct 31 00:14:19.887405 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:19.887382 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-2"] Oct 31 00:14:19.902908 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:19.902877 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:14:19.911289 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:19.911254 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpff8\" (UniqueName: \"kubernetes.io/projected/e7e2a5db-fbe1-4127-ab3f-98be037db950-kube-api-access-fpff8\") pod \"data-science-collector-collector-2\" (UID: \"e7e2a5db-fbe1-4127-ab3f-98be037db950\") " pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 31 00:14:19.911411 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:19.911357 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7e2a5db-fbe1-4127-ab3f-98be037db950-tls-certs\") pod \"data-science-collector-collector-2\" (UID: \"e7e2a5db-fbe1-4127-ab3f-98be037db950\") " pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 31 00:14:19.911474 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:19.911420 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7e2a5db-fbe1-4127-ab3f-98be037db950-otlp-tls-certs\") pod \"data-science-collector-collector-2\" (UID: \"e7e2a5db-fbe1-4127-ab3f-98be037db950\") " pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 31 00:14:19.911474 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:19.911454 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e7e2a5db-fbe1-4127-ab3f-98be037db950-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-2\" (UID: \"e7e2a5db-fbe1-4127-ab3f-98be037db950\") " pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 31 00:14:19.911543 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:19.911508 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e7e2a5db-fbe1-4127-ab3f-98be037db950-otc-internal\") pod \"data-science-collector-collector-2\" (UID: \"e7e2a5db-fbe1-4127-ab3f-98be037db950\") " pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 31 00:14:19.911584 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:19.911544 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7e2a5db-fbe1-4127-ab3f-98be037db950-monitoring-tls-certs\") pod \"data-science-collector-collector-2\" (UID: \"e7e2a5db-fbe1-4127-ab3f-98be037db950\") " pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 31 00:14:19.982371 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:19.982340 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:14:20.013060 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:20.013024 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e7e2a5db-fbe1-4127-ab3f-98be037db950-otc-internal\") pod \"data-science-collector-collector-2\" (UID: \"e7e2a5db-fbe1-4127-ab3f-98be037db950\") " pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 31 00:14:20.013060 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:20.013061 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7e2a5db-fbe1-4127-ab3f-98be037db950-monitoring-tls-certs\") pod \"data-science-collector-collector-2\" (UID: \"e7e2a5db-fbe1-4127-ab3f-98be037db950\") " pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 31 00:14:20.013275 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:20.013161 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpff8\" (UniqueName: \"kubernetes.io/projected/e7e2a5db-fbe1-4127-ab3f-98be037db950-kube-api-access-fpff8\") pod \"data-science-collector-collector-2\" (UID: \"e7e2a5db-fbe1-4127-ab3f-98be037db950\") " pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 31 00:14:20.013275 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:20.013199 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7e2a5db-fbe1-4127-ab3f-98be037db950-tls-certs\") pod \"data-science-collector-collector-2\" (UID: \"e7e2a5db-fbe1-4127-ab3f-98be037db950\") " pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 31 00:14:20.013275 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:20.013237 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7e2a5db-fbe1-4127-ab3f-98be037db950-otlp-tls-certs\") pod \"data-science-collector-collector-2\" (UID: \"e7e2a5db-fbe1-4127-ab3f-98be037db950\") " pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 31 00:14:20.013402 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:20.013366 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e7e2a5db-fbe1-4127-ab3f-98be037db950-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-2\" (UID: \"e7e2a5db-fbe1-4127-ab3f-98be037db950\") " pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 31 00:14:20.013885 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:20.013858 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e7e2a5db-fbe1-4127-ab3f-98be037db950-otc-internal\") pod \"data-science-collector-collector-2\" (UID: \"e7e2a5db-fbe1-4127-ab3f-98be037db950\") " pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 31 00:14:20.015769 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:20.015745 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7e2a5db-fbe1-4127-ab3f-98be037db950-tls-certs\") pod \"data-science-collector-collector-2\" (UID: \"e7e2a5db-fbe1-4127-ab3f-98be037db950\") " pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 31 00:14:20.015879 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:20.015802 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e7e2a5db-fbe1-4127-ab3f-98be037db950-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-2\" (UID: \"e7e2a5db-fbe1-4127-ab3f-98be037db950\") " pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 31 00:14:20.015879 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:20.015855 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7e2a5db-fbe1-4127-ab3f-98be037db950-monitoring-tls-certs\") pod \"data-science-collector-collector-2\" (UID: \"e7e2a5db-fbe1-4127-ab3f-98be037db950\") " pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 31 00:14:20.015965 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:20.015925 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7e2a5db-fbe1-4127-ab3f-98be037db950-otlp-tls-certs\") pod \"data-science-collector-collector-2\" (UID: \"e7e2a5db-fbe1-4127-ab3f-98be037db950\") " pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 31 00:14:20.025541 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:20.025515 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpff8\" (UniqueName: \"kubernetes.io/projected/e7e2a5db-fbe1-4127-ab3f-98be037db950-kube-api-access-fpff8\") pod \"data-science-collector-collector-2\" (UID: \"e7e2a5db-fbe1-4127-ab3f-98be037db950\") " pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 31 00:14:20.189712 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:20.189624 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 31 00:14:20.466131 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:20.466090 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-2"] Oct 31 00:14:20.468463 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:14:20.468433 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7e2a5db_fbe1_4127_ab3f_98be037db950.slice/crio-5df7a911f9374d32c8af94db42a45726e387081ebdc2f1af28579656a495fc4d WatchSource:0}: Error finding container 5df7a911f9374d32c8af94db42a45726e387081ebdc2f1af28579656a495fc4d: Status 404 returned error can't find the container with id 5df7a911f9374d32c8af94db42a45726e387081ebdc2f1af28579656a495fc4d Oct 31 00:14:20.470455 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:20.470436 5274 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:14:20.899412 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:14:20.899377 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:14:20.937713 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:20.937678 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-2" event={"ID":"e7e2a5db-fbe1-4127-ab3f-98be037db950","Type":"ContainerStarted","Data":"5df7a911f9374d32c8af94db42a45726e387081ebdc2f1af28579656a495fc4d"} Oct 31 00:14:25.054446 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:25.054406 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:14:25.054767 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:25.054469 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:14:25.966526 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:25.966491 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-2" event={"ID":"e7e2a5db-fbe1-4127-ab3f-98be037db950","Type":"ContainerStarted","Data":"6336855f63fca9be3d7f0e45a625806cf803c756a9e7612eb25caf3465546c9b"} Oct 31 00:14:25.988602 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:25.988540 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-2" podStartSLOduration=2.405602867 podStartE2EDuration="6.988521755s" totalImagesPullingTime="4.582918888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:25.987263525 +0000 UTC m=+902.624082242" watchObservedRunningTime="2026-10-31 00:14:25.988521755 +0000 UTC m=+902.625340470" Oct 31 00:14:31.920409 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:31.920370 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-2"] Oct 31 00:14:31.921087 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:31.920620 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-2" podUID="e7e2a5db-fbe1-4127-ab3f-98be037db950" containerName="otc-container" containerID="cri-o://6336855f63fca9be3d7f0e45a625806cf803c756a9e7612eb25caf3465546c9b" gracePeriod=30 Oct 31 00:14:33.175926 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:33.175902 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 31 00:14:33.240713 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:33.240676 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7e2a5db-fbe1-4127-ab3f-98be037db950-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7e2a5db-fbe1-4127-ab3f-98be037db950-otlp-tls-certs\") pod \"e7e2a5db-fbe1-4127-ab3f-98be037db950\" (UID: \"e7e2a5db-fbe1-4127-ab3f-98be037db950\") " Oct 31 00:14:33.240888 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:33.240730 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7e2a5db-fbe1-4127-ab3f-98be037db950-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e7e2a5db-fbe1-4127-ab3f-98be037db950-data-science-collector-ta-client-cert\") pod \"e7e2a5db-fbe1-4127-ab3f-98be037db950\" (UID: \"e7e2a5db-fbe1-4127-ab3f-98be037db950\") " Oct 31 00:14:33.240888 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:33.240849 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7e2a5db-fbe1-4127-ab3f-98be037db950-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7e2a5db-fbe1-4127-ab3f-98be037db950-monitoring-tls-certs\") pod \"e7e2a5db-fbe1-4127-ab3f-98be037db950\" (UID: \"e7e2a5db-fbe1-4127-ab3f-98be037db950\") " Oct 31 00:14:33.241001 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:33.240887 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7e2a5db-fbe1-4127-ab3f-98be037db950-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7e2a5db-fbe1-4127-ab3f-98be037db950-tls-certs\") pod \"e7e2a5db-fbe1-4127-ab3f-98be037db950\" (UID: \"e7e2a5db-fbe1-4127-ab3f-98be037db950\") " Oct 31 00:14:33.241001 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:33.240945 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7e2a5db-fbe1-4127-ab3f-98be037db950-otc-internal\" (UniqueName: \"kubernetes.io/configmap/e7e2a5db-fbe1-4127-ab3f-98be037db950-otc-internal\") pod \"e7e2a5db-fbe1-4127-ab3f-98be037db950\" (UID: \"e7e2a5db-fbe1-4127-ab3f-98be037db950\") " Oct 31 00:14:33.241120 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:33.241001 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7e2a5db-fbe1-4127-ab3f-98be037db950-kube-api-access-fpff8\" (UniqueName: \"kubernetes.io/projected/e7e2a5db-fbe1-4127-ab3f-98be037db950-kube-api-access-fpff8\") pod \"e7e2a5db-fbe1-4127-ab3f-98be037db950\" (UID: \"e7e2a5db-fbe1-4127-ab3f-98be037db950\") " Oct 31 00:14:33.241422 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:33.241387 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7e2a5db-fbe1-4127-ab3f-98be037db950-otc-internal" pod "e7e2a5db-fbe1-4127-ab3f-98be037db950" (UID: "e7e2a5db-fbe1-4127-ab3f-98be037db950"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:14:33.243025 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:33.242998 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7e2a5db-fbe1-4127-ab3f-98be037db950-otlp-tls-certs" pod "e7e2a5db-fbe1-4127-ab3f-98be037db950" (UID: "e7e2a5db-fbe1-4127-ab3f-98be037db950"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:33.243206 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:33.243027 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7e2a5db-fbe1-4127-ab3f-98be037db950-tls-certs" pod "e7e2a5db-fbe1-4127-ab3f-98be037db950" (UID: "e7e2a5db-fbe1-4127-ab3f-98be037db950"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:33.243206 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:33.243134 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7e2a5db-fbe1-4127-ab3f-98be037db950-kube-api-access-fpff8" pod "e7e2a5db-fbe1-4127-ab3f-98be037db950" (UID: "e7e2a5db-fbe1-4127-ab3f-98be037db950"). InnerVolumeSpecName "kube-api-access-fpff8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:14:33.243390 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:33.243374 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7e2a5db-fbe1-4127-ab3f-98be037db950-data-science-collector-ta-client-cert" pod "e7e2a5db-fbe1-4127-ab3f-98be037db950" (UID: "e7e2a5db-fbe1-4127-ab3f-98be037db950"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:33.243439 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:33.243393 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7e2a5db-fbe1-4127-ab3f-98be037db950-monitoring-tls-certs" pod "e7e2a5db-fbe1-4127-ab3f-98be037db950" (UID: "e7e2a5db-fbe1-4127-ab3f-98be037db950"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:33.342238 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:33.342200 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpff8\" (UniqueName: \"kubernetes.io/projected/e7e2a5db-fbe1-4127-ab3f-98be037db950-kube-api-access-fpff8\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:33.342238 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:33.342228 5274 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7e2a5db-fbe1-4127-ab3f-98be037db950-otlp-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:33.342238 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:33.342241 5274 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e7e2a5db-fbe1-4127-ab3f-98be037db950-data-science-collector-ta-client-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:33.342487 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:33.342251 5274 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7e2a5db-fbe1-4127-ab3f-98be037db950-monitoring-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:33.342487 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:33.342260 5274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7e2a5db-fbe1-4127-ab3f-98be037db950-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:33.342487 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:33.342269 5274 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e7e2a5db-fbe1-4127-ab3f-98be037db950-otc-internal\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:34.009920 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:34.009882 5274 generic.go:356] "Generic (PLEG): container finished" podID="e7e2a5db-fbe1-4127-ab3f-98be037db950" containerID="6336855f63fca9be3d7f0e45a625806cf803c756a9e7612eb25caf3465546c9b" exitCode=0 Oct 31 00:14:34.010116 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:34.009937 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-2" Oct 31 00:14:34.010116 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:34.009977 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-2" event={"ID":"e7e2a5db-fbe1-4127-ab3f-98be037db950","Type":"ContainerDied","Data":"6336855f63fca9be3d7f0e45a625806cf803c756a9e7612eb25caf3465546c9b"} Oct 31 00:14:34.010116 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:34.010028 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-2" event={"ID":"e7e2a5db-fbe1-4127-ab3f-98be037db950","Type":"ContainerDied","Data":"5df7a911f9374d32c8af94db42a45726e387081ebdc2f1af28579656a495fc4d"} Oct 31 00:14:34.010116 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:34.010049 5274 scope.go:122] "RemoveContainer" containerID="6336855f63fca9be3d7f0e45a625806cf803c756a9e7612eb25caf3465546c9b" Oct 31 00:14:34.020006 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:34.019975 5274 scope.go:122] "RemoveContainer" containerID="6336855f63fca9be3d7f0e45a625806cf803c756a9e7612eb25caf3465546c9b" Oct 31 00:14:34.020349 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:14:34.020327 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6336855f63fca9be3d7f0e45a625806cf803c756a9e7612eb25caf3465546c9b\": container with ID starting with 6336855f63fca9be3d7f0e45a625806cf803c756a9e7612eb25caf3465546c9b not found: ID does not exist" containerID="6336855f63fca9be3d7f0e45a625806cf803c756a9e7612eb25caf3465546c9b" Oct 31 00:14:34.020429 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:34.020372 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6336855f63fca9be3d7f0e45a625806cf803c756a9e7612eb25caf3465546c9b"} err="failed to get container status \"6336855f63fca9be3d7f0e45a625806cf803c756a9e7612eb25caf3465546c9b\": rpc error: code = NotFound desc = could not find container \"6336855f63fca9be3d7f0e45a625806cf803c756a9e7612eb25caf3465546c9b\": container with ID starting with 6336855f63fca9be3d7f0e45a625806cf803c756a9e7612eb25caf3465546c9b not found: ID does not exist" Oct 31 00:14:34.030980 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:34.030935 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-2"] Oct 31 00:14:34.035450 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:34.035428 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-2"] Oct 31 00:14:35.481072 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:35.481042 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7e2a5db-fbe1-4127-ab3f-98be037db950" path="/var/lib/kubelet/pods/e7e2a5db-fbe1-4127-ab3f-98be037db950/volumes" Oct 31 00:14:50.900379 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:14:50.900336 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:14:52.815431 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:52.815397 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jp79k"] Oct 31 00:14:52.815942 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:52.815778 5274 cpu_manager.go:414] "containerMap: removing container" podUID="e7e2a5db-fbe1-4127-ab3f-98be037db950" containerName="otc-container" Oct 31 00:14:52.815942 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:52.815791 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7e2a5db-fbe1-4127-ab3f-98be037db950" containerName="otc-container" Oct 31 00:14:52.815942 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:52.815864 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7e2a5db-fbe1-4127-ab3f-98be037db950" containerName="otc-container" Oct 31 00:14:52.818817 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:52.818795 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jp79k" Oct 31 00:14:52.828053 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:52.828027 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"tests-monitoring-tenant-metrics\"/\"openshift-service-ca.crt\"" Oct 31 00:14:52.830220 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:52.830196 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"tests-monitoring-tenant-metrics\"/\"default-dockercfg-ncjsr\"" Oct 31 00:14:52.830317 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:52.830296 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"tests-monitoring-tenant-metrics\"/\"kube-root-ca.crt\"" Oct 31 00:14:52.830606 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:52.830586 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jp79k"] Oct 31 00:14:52.920002 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:52.919935 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxqcv\" (UniqueName: \"kubernetes.io/projected/3128efee-fcf5-43cf-be0c-9388edf25085-kube-api-access-dxqcv\") pod \"tenant-metrics-exporter-5cf544fb97-jp79k\" (UID: \"3128efee-fcf5-43cf-be0c-9388edf25085\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jp79k" Oct 31 00:14:52.920182 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:52.920075 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3128efee-fcf5-43cf-be0c-9388edf25085-data\") pod \"tenant-metrics-exporter-5cf544fb97-jp79k\" (UID: \"3128efee-fcf5-43cf-be0c-9388edf25085\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jp79k" Oct 31 00:14:53.021516 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:53.021482 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxqcv\" (UniqueName: \"kubernetes.io/projected/3128efee-fcf5-43cf-be0c-9388edf25085-kube-api-access-dxqcv\") pod \"tenant-metrics-exporter-5cf544fb97-jp79k\" (UID: \"3128efee-fcf5-43cf-be0c-9388edf25085\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jp79k" Oct 31 00:14:53.021683 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:53.021531 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3128efee-fcf5-43cf-be0c-9388edf25085-data\") pod \"tenant-metrics-exporter-5cf544fb97-jp79k\" (UID: \"3128efee-fcf5-43cf-be0c-9388edf25085\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jp79k" Oct 31 00:14:53.021975 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:53.021931 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3128efee-fcf5-43cf-be0c-9388edf25085-data\") pod \"tenant-metrics-exporter-5cf544fb97-jp79k\" (UID: \"3128efee-fcf5-43cf-be0c-9388edf25085\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jp79k" Oct 31 00:14:53.036409 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:53.036384 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxqcv\" (UniqueName: \"kubernetes.io/projected/3128efee-fcf5-43cf-be0c-9388edf25085-kube-api-access-dxqcv\") pod \"tenant-metrics-exporter-5cf544fb97-jp79k\" (UID: \"3128efee-fcf5-43cf-be0c-9388edf25085\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jp79k" Oct 31 00:14:53.134036 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:53.134009 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jp79k" Oct 31 00:14:53.404679 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:53.404653 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jp79k"] Oct 31 00:14:53.406537 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:14:53.406510 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3128efee_fcf5_43cf_be0c_9388edf25085.slice/crio-a9c059be685f515fa42b8c44d81c4cb0f1451d46c98a2b44881b98d22cd46bda WatchSource:0}: Error finding container a9c059be685f515fa42b8c44d81c4cb0f1451d46c98a2b44881b98d22cd46bda: Status 404 returned error can't find the container with id a9c059be685f515fa42b8c44d81c4cb0f1451d46c98a2b44881b98d22cd46bda Oct 31 00:14:54.113586 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:54.113555 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jp79k" event={"ID":"3128efee-fcf5-43cf-be0c-9388edf25085","Type":"ContainerStarted","Data":"a9c059be685f515fa42b8c44d81c4cb0f1451d46c98a2b44881b98d22cd46bda"} Oct 31 00:14:58.145342 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:58.145309 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jp79k" event={"ID":"3128efee-fcf5-43cf-be0c-9388edf25085","Type":"ContainerStarted","Data":"656a49c3e17da143b3af29c7f3d7c359391e997855a0107c71a7929ae2a13c49"} Oct 31 00:14:58.162230 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:14:58.162184 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jp79k" podStartSLOduration=2.482501632 podStartE2EDuration="6.1621696s" totalImagesPullingTime="3.679667968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:58.161189994 +0000 UTC m=+934.798008692" watchObservedRunningTime="2026-10-31 00:14:58.1621696 +0000 UTC m=+934.798988316" Oct 31 00:15:20.899123 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:15:20.899086 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:15:50.899470 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:15:50.899427 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:16:14.786645 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:14.786570 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jp79k"] Oct 31 00:16:14.787132 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:14.786752 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jp79k" podUID="3128efee-fcf5-43cf-be0c-9388edf25085" containerName="tenant-metrics-exporter" containerID="cri-o://656a49c3e17da143b3af29c7f3d7c359391e997855a0107c71a7929ae2a13c49" gracePeriod=30 Oct 31 00:16:15.026687 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:15.026664 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jp79k" Oct 31 00:16:15.105537 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:15.105463 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3128efee-fcf5-43cf-be0c-9388edf25085-kube-api-access-dxqcv\" (UniqueName: \"kubernetes.io/projected/3128efee-fcf5-43cf-be0c-9388edf25085-kube-api-access-dxqcv\") pod \"3128efee-fcf5-43cf-be0c-9388edf25085\" (UID: \"3128efee-fcf5-43cf-be0c-9388edf25085\") " Oct 31 00:16:15.105537 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:15.105530 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3128efee-fcf5-43cf-be0c-9388edf25085-data\" (UniqueName: \"kubernetes.io/empty-dir/3128efee-fcf5-43cf-be0c-9388edf25085-data\") pod \"3128efee-fcf5-43cf-be0c-9388edf25085\" (UID: \"3128efee-fcf5-43cf-be0c-9388edf25085\") " Oct 31 00:16:15.105927 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:15.105899 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3128efee-fcf5-43cf-be0c-9388edf25085-data" pod "3128efee-fcf5-43cf-be0c-9388edf25085" (UID: "3128efee-fcf5-43cf-be0c-9388edf25085"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:16:15.107692 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:15.107672 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3128efee-fcf5-43cf-be0c-9388edf25085-kube-api-access-dxqcv" pod "3128efee-fcf5-43cf-be0c-9388edf25085" (UID: "3128efee-fcf5-43cf-be0c-9388edf25085"). InnerVolumeSpecName "kube-api-access-dxqcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:16:15.206413 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:15.206391 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxqcv\" (UniqueName: \"kubernetes.io/projected/3128efee-fcf5-43cf-be0c-9388edf25085-kube-api-access-dxqcv\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:15.206413 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:15.206410 5274 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3128efee-fcf5-43cf-be0c-9388edf25085-data\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:15.533457 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:15.533427 5274 generic.go:356] "Generic (PLEG): container finished" podID="3128efee-fcf5-43cf-be0c-9388edf25085" containerID="656a49c3e17da143b3af29c7f3d7c359391e997855a0107c71a7929ae2a13c49" exitCode=0 Oct 31 00:16:15.533601 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:15.533472 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jp79k" Oct 31 00:16:15.533601 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:15.533510 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jp79k" event={"ID":"3128efee-fcf5-43cf-be0c-9388edf25085","Type":"ContainerDied","Data":"656a49c3e17da143b3af29c7f3d7c359391e997855a0107c71a7929ae2a13c49"} Oct 31 00:16:15.533601 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:15.533564 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jp79k" event={"ID":"3128efee-fcf5-43cf-be0c-9388edf25085","Type":"ContainerDied","Data":"a9c059be685f515fa42b8c44d81c4cb0f1451d46c98a2b44881b98d22cd46bda"} Oct 31 00:16:15.533601 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:15.533587 5274 scope.go:122] "RemoveContainer" containerID="656a49c3e17da143b3af29c7f3d7c359391e997855a0107c71a7929ae2a13c49" Oct 31 00:16:15.542074 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:15.541988 5274 scope.go:122] "RemoveContainer" containerID="656a49c3e17da143b3af29c7f3d7c359391e997855a0107c71a7929ae2a13c49" Oct 31 00:16:15.542230 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:16:15.542213 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"656a49c3e17da143b3af29c7f3d7c359391e997855a0107c71a7929ae2a13c49\": container with ID starting with 656a49c3e17da143b3af29c7f3d7c359391e997855a0107c71a7929ae2a13c49 not found: ID does not exist" containerID="656a49c3e17da143b3af29c7f3d7c359391e997855a0107c71a7929ae2a13c49" Oct 31 00:16:15.542281 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:15.542240 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"656a49c3e17da143b3af29c7f3d7c359391e997855a0107c71a7929ae2a13c49"} err="failed to get container status \"656a49c3e17da143b3af29c7f3d7c359391e997855a0107c71a7929ae2a13c49\": rpc error: code = NotFound desc = could not find container \"656a49c3e17da143b3af29c7f3d7c359391e997855a0107c71a7929ae2a13c49\": container with ID starting with 656a49c3e17da143b3af29c7f3d7c359391e997855a0107c71a7929ae2a13c49 not found: ID does not exist" Oct 31 00:16:15.551135 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:15.551112 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jp79k"] Oct 31 00:16:15.557396 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:15.557372 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-jp79k"] Oct 31 00:16:17.480738 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:17.480701 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3128efee-fcf5-43cf-be0c-9388edf25085" path="/var/lib/kubelet/pods/3128efee-fcf5-43cf-be0c-9388edf25085/volumes" Oct 31 00:16:20.898836 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:16:20.898800 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:16:29.204536 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:29.204502 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["tests-monitoring-dcgm-https-18e37510356f4af5/dcgm-https-exporter"] Oct 31 00:16:29.204939 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:29.204925 5274 cpu_manager.go:414] "containerMap: removing container" podUID="3128efee-fcf5-43cf-be0c-9388edf25085" containerName="tenant-metrics-exporter" Oct 31 00:16:29.205033 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:29.204941 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3128efee-fcf5-43cf-be0c-9388edf25085" containerName="tenant-metrics-exporter" Oct 31 00:16:29.205078 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:29.205031 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="3128efee-fcf5-43cf-be0c-9388edf25085" containerName="tenant-metrics-exporter" Oct 31 00:16:29.209397 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:29.209382 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-dcgm-https-18e37510356f4af5/dcgm-https-exporter" Oct 31 00:16:29.215587 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:29.215563 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"tests-monitoring-dcgm-https-18e37510356f4af5\"/\"dcgm-https-exporter\"" Oct 31 00:16:29.215789 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:29.215749 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"tests-monitoring-dcgm-https-18e37510356f4af5\"/\"dcgm-https-tls\"" Oct 31 00:16:29.215888 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:29.215827 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"tests-monitoring-dcgm-https-18e37510356f4af5\"/\"default-dockercfg-cvv4r\"" Oct 31 00:16:29.216999 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:29.216818 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["tests-monitoring-dcgm-https-18e37510356f4af5/dcgm-https-exporter"] Oct 31 00:16:29.329979 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:29.329904 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/010ef2cc-5f12-48eb-a708-d6418e0da0af-tls\") pod \"dcgm-https-exporter\" (UID: \"010ef2cc-5f12-48eb-a708-d6418e0da0af\") " pod="tests-monitoring-dcgm-https-18e37510356f4af5/dcgm-https-exporter" Oct 31 00:16:29.330157 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:29.330038 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/010ef2cc-5f12-48eb-a708-d6418e0da0af-data\") pod \"dcgm-https-exporter\" (UID: \"010ef2cc-5f12-48eb-a708-d6418e0da0af\") " pod="tests-monitoring-dcgm-https-18e37510356f4af5/dcgm-https-exporter" Oct 31 00:16:29.330157 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:29.330071 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/configmap/010ef2cc-5f12-48eb-a708-d6418e0da0af-web-config\") pod \"dcgm-https-exporter\" (UID: \"010ef2cc-5f12-48eb-a708-d6418e0da0af\") " pod="tests-monitoring-dcgm-https-18e37510356f4af5/dcgm-https-exporter" Oct 31 00:16:29.430664 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:29.430627 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/010ef2cc-5f12-48eb-a708-d6418e0da0af-tls\") pod \"dcgm-https-exporter\" (UID: \"010ef2cc-5f12-48eb-a708-d6418e0da0af\") " pod="tests-monitoring-dcgm-https-18e37510356f4af5/dcgm-https-exporter" Oct 31 00:16:29.430813 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:29.430679 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/010ef2cc-5f12-48eb-a708-d6418e0da0af-data\") pod \"dcgm-https-exporter\" (UID: \"010ef2cc-5f12-48eb-a708-d6418e0da0af\") " pod="tests-monitoring-dcgm-https-18e37510356f4af5/dcgm-https-exporter" Oct 31 00:16:29.430813 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:29.430708 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/configmap/010ef2cc-5f12-48eb-a708-d6418e0da0af-web-config\") pod \"dcgm-https-exporter\" (UID: \"010ef2cc-5f12-48eb-a708-d6418e0da0af\") " pod="tests-monitoring-dcgm-https-18e37510356f4af5/dcgm-https-exporter" Oct 31 00:16:29.431077 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:29.431057 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/010ef2cc-5f12-48eb-a708-d6418e0da0af-data\") pod \"dcgm-https-exporter\" (UID: \"010ef2cc-5f12-48eb-a708-d6418e0da0af\") " pod="tests-monitoring-dcgm-https-18e37510356f4af5/dcgm-https-exporter" Oct 31 00:16:29.431523 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:29.431493 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/configmap/010ef2cc-5f12-48eb-a708-d6418e0da0af-web-config\") pod \"dcgm-https-exporter\" (UID: \"010ef2cc-5f12-48eb-a708-d6418e0da0af\") " pod="tests-monitoring-dcgm-https-18e37510356f4af5/dcgm-https-exporter" Oct 31 00:16:29.433253 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:29.433235 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/010ef2cc-5f12-48eb-a708-d6418e0da0af-tls\") pod \"dcgm-https-exporter\" (UID: \"010ef2cc-5f12-48eb-a708-d6418e0da0af\") " pod="tests-monitoring-dcgm-https-18e37510356f4af5/dcgm-https-exporter" Oct 31 00:16:29.523674 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:29.523606 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-dcgm-https-18e37510356f4af5/dcgm-https-exporter" Oct 31 00:16:29.795646 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:29.795621 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["tests-monitoring-dcgm-https-18e37510356f4af5/dcgm-https-exporter"] Oct 31 00:16:29.797634 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:16:29.797600 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod010ef2cc_5f12_48eb_a708_d6418e0da0af.slice/crio-6f6fab66ccca022868e0925ba6afc75a2547668cf76d5bbf43fffcd300ed8478 WatchSource:0}: Error finding container 6f6fab66ccca022868e0925ba6afc75a2547668cf76d5bbf43fffcd300ed8478: Status 404 returned error can't find the container with id 6f6fab66ccca022868e0925ba6afc75a2547668cf76d5bbf43fffcd300ed8478 Oct 31 00:16:30.606286 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:30.606250 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-dcgm-https-18e37510356f4af5/dcgm-https-exporter" event={"ID":"010ef2cc-5f12-48eb-a708-d6418e0da0af","Type":"ContainerStarted","Data":"405104e3721140cad624bec9409f0b469adfffe2b10210eb5c9d08244567d334"} Oct 31 00:16:30.606685 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:30.606298 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-dcgm-https-18e37510356f4af5/dcgm-https-exporter" event={"ID":"010ef2cc-5f12-48eb-a708-d6418e0da0af","Type":"ContainerStarted","Data":"6f6fab66ccca022868e0925ba6afc75a2547668cf76d5bbf43fffcd300ed8478"} Oct 31 00:16:30.606685 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:30.606344 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="tests-monitoring-dcgm-https-18e37510356f4af5/dcgm-https-exporter" Oct 31 00:16:30.625273 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:30.625231 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="tests-monitoring-dcgm-https-18e37510356f4af5/dcgm-https-exporter" podStartSLOduration=1.625219615 podStartE2EDuration="1.625219615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:30.62382635 +0000 UTC m=+1027.260645065" watchObservedRunningTime="2026-10-31 00:16:30.625219615 +0000 UTC m=+1027.262038331" Oct 31 00:16:36.613852 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:36.613817 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="tests-monitoring-dcgm-https-18e37510356f4af5/dcgm-https-exporter" Oct 31 00:16:50.899453 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:16:50.899417 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:16:54.702638 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:54.702603 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:16:54.706203 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:54.706187 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:16:54.713262 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:54.713241 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:16:54.715697 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:54.715679 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-g8x7h\"" Oct 31 00:16:54.718701 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:54.718678 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:16:54.718808 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:54.718796 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-db539ca7\"" Oct 31 00:16:54.813537 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:54.813510 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:16:54.813728 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:54.813551 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:16:54.813728 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:54.813674 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:16:54.855734 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:54.855707 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:16:54.855843 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:54.855741 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:16:54.855843 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:54.855765 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:16:54.855843 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:54.855833 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:16:54.855949 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:54.855932 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:16:54.856007 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:54.855983 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l728k\" (UniqueName: \"kubernetes.io/projected/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-kube-api-access-l728k\") pod \"data-science-collector-collector-0\" (UID: \"cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:16:54.956740 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:54.956686 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:16:54.956740 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:54.956735 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:16:54.956905 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:54.956754 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l728k\" (UniqueName: \"kubernetes.io/projected/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-kube-api-access-l728k\") pod \"data-science-collector-collector-0\" (UID: \"cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:16:54.956905 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:54.956797 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:16:54.956905 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:54.956820 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:16:54.956905 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:54.956839 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:16:54.957426 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:54.957394 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:16:54.959360 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:54.959329 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:16:54.959360 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:54.959347 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:16:54.959713 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:54.959692 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:16:54.959802 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:54.959787 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:16:54.969689 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:54.969666 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l728k\" (UniqueName: \"kubernetes.io/projected/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-kube-api-access-l728k\") pod \"data-science-collector-collector-0\" (UID: \"cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:16:55.023905 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:55.023887 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:16:55.298420 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:55.298390 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:16:55.300230 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:16:55.300204 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb8ea2b6_ad7c_4646_8a2b_ceda7bcaa246.slice/crio-b1ec6e439f7f7807eb8feddfccc7b14ac5eeb8ca52b934dccfaa919ec7ef757d WatchSource:0}: Error finding container b1ec6e439f7f7807eb8feddfccc7b14ac5eeb8ca52b934dccfaa919ec7ef757d: Status 404 returned error can't find the container with id b1ec6e439f7f7807eb8feddfccc7b14ac5eeb8ca52b934dccfaa919ec7ef757d Oct 31 00:16:55.302391 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:55.302357 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:16:55.302487 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:55.302425 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:16:55.738220 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:55.738184 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246","Type":"ContainerStarted","Data":"4a2962994ed2b9ff68cb5a02374bffa77da04dd75bef522329246d36a5891af4"} Oct 31 00:16:55.738607 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:55.738239 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246","Type":"ContainerStarted","Data":"b1ec6e439f7f7807eb8feddfccc7b14ac5eeb8ca52b934dccfaa919ec7ef757d"} Oct 31 00:16:55.760600 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:16:55.760554 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.760539941 podStartE2EDuration="1.760539941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:55.758328108 +0000 UTC m=+1052.395146825" watchObservedRunningTime="2026-10-31 00:16:55.760539941 +0000 UTC m=+1052.397358657" Oct 31 00:17:20.898554 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:17:20.898518 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:17:50.899006 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:17:50.898967 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:18:20.898769 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:18:20.898728 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:18:50.899782 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:18:50.899745 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:19:16.296755 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:16.296540 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:19:16.297397 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:16.296826 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246" containerName="otc-container" containerID="cri-o://4a2962994ed2b9ff68cb5a02374bffa77da04dd75bef522329246d36a5891af4" gracePeriod=30 Oct 31 00:19:18.650367 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:18.650342 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:18.726866 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:18.726780 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-data-science-collector-ta-client-cert\") pod \"cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246\" (UID: \"cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246\") " Oct 31 00:19:18.726866 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:18.726854 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-tls-certs\") pod \"cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246\" (UID: \"cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246\") " Oct 31 00:19:18.727135 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:18.726881 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-otc-internal\" (UniqueName: \"kubernetes.io/configmap/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-otc-internal\") pod \"cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246\" (UID: \"cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246\") " Oct 31 00:19:18.727135 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:18.726940 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-kube-api-access-l728k\" (UniqueName: \"kubernetes.io/projected/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-kube-api-access-l728k\") pod \"cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246\" (UID: \"cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246\") " Oct 31 00:19:18.727135 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:18.727009 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-monitoring-tls-certs\") pod \"cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246\" (UID: \"cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246\") " Oct 31 00:19:18.727135 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:18.727029 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-otlp-tls-certs\") pod \"cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246\" (UID: \"cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246\") " Oct 31 00:19:18.727384 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:18.727353 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-otc-internal" pod "cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246" (UID: "cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:19:18.729171 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:18.729139 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-otlp-tls-certs" pod "cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246" (UID: "cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:18.729279 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:18.729197 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-data-science-collector-ta-client-cert" pod "cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246" (UID: "cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:18.729279 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:18.729204 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-kube-api-access-l728k" pod "cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246" (UID: "cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246"). InnerVolumeSpecName "kube-api-access-l728k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:19:18.729437 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:18.729415 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-monitoring-tls-certs" pod "cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246" (UID: "cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:18.729543 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:18.729432 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-tls-certs" pod "cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246" (UID: "cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:18.827685 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:18.827658 5274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:18.827685 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:18.827682 5274 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-otc-internal\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:18.827878 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:18.827694 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l728k\" (UniqueName: \"kubernetes.io/projected/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-kube-api-access-l728k\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:18.827878 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:18.827704 5274 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-monitoring-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:18.827878 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:18.827713 5274 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-otlp-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:18.827878 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:18.827724 5274 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246-data-science-collector-ta-client-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:18.879220 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:18.879192 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["tests-monitoring-dcgm-https-18e37510356f4af5/dcgm-https-exporter"] Oct 31 00:19:18.879438 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:18.879404 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="tests-monitoring-dcgm-https-18e37510356f4af5/dcgm-https-exporter" podUID="010ef2cc-5f12-48eb-a708-d6418e0da0af" containerName="dcgm-https-exporter" containerID="cri-o://405104e3721140cad624bec9409f0b469adfffe2b10210eb5c9d08244567d334" gracePeriod=30 Oct 31 00:19:19.122408 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.122373 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-dcgm-https-18e37510356f4af5/dcgm-https-exporter" Oct 31 00:19:19.129477 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.129457 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/010ef2cc-5f12-48eb-a708-d6418e0da0af-web-config\" (UniqueName: \"kubernetes.io/configmap/010ef2cc-5f12-48eb-a708-d6418e0da0af-web-config\") pod \"010ef2cc-5f12-48eb-a708-d6418e0da0af\" (UID: \"010ef2cc-5f12-48eb-a708-d6418e0da0af\") " Oct 31 00:19:19.129557 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.129500 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/010ef2cc-5f12-48eb-a708-d6418e0da0af-data\" (UniqueName: \"kubernetes.io/empty-dir/010ef2cc-5f12-48eb-a708-d6418e0da0af-data\") pod \"010ef2cc-5f12-48eb-a708-d6418e0da0af\" (UID: \"010ef2cc-5f12-48eb-a708-d6418e0da0af\") " Oct 31 00:19:19.129557 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.129538 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/010ef2cc-5f12-48eb-a708-d6418e0da0af-tls\" (UniqueName: \"kubernetes.io/secret/010ef2cc-5f12-48eb-a708-d6418e0da0af-tls\") pod \"010ef2cc-5f12-48eb-a708-d6418e0da0af\" (UID: \"010ef2cc-5f12-48eb-a708-d6418e0da0af\") " Oct 31 00:19:19.129945 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.129912 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/010ef2cc-5f12-48eb-a708-d6418e0da0af-data" pod "010ef2cc-5f12-48eb-a708-d6418e0da0af" (UID: "010ef2cc-5f12-48eb-a708-d6418e0da0af"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:19:19.131526 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.131508 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/010ef2cc-5f12-48eb-a708-d6418e0da0af-tls" pod "010ef2cc-5f12-48eb-a708-d6418e0da0af" (UID: "010ef2cc-5f12-48eb-a708-d6418e0da0af"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:19.134304 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.134281 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/010ef2cc-5f12-48eb-a708-d6418e0da0af-web-config" pod "010ef2cc-5f12-48eb-a708-d6418e0da0af" (UID: "010ef2cc-5f12-48eb-a708-d6418e0da0af"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:19:19.230801 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.230770 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/010ef2cc-5f12-48eb-a708-d6418e0da0af-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:19.230801 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.230795 5274 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/configmap/010ef2cc-5f12-48eb-a708-d6418e0da0af-web-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:19.230801 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.230804 5274 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/010ef2cc-5f12-48eb-a708-d6418e0da0af-data\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:19.448026 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.447991 5274 generic.go:356] "Generic (PLEG): container finished" podID="010ef2cc-5f12-48eb-a708-d6418e0da0af" containerID="405104e3721140cad624bec9409f0b469adfffe2b10210eb5c9d08244567d334" exitCode=0 Oct 31 00:19:19.448182 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.448036 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-dcgm-https-18e37510356f4af5/dcgm-https-exporter" Oct 31 00:19:19.448182 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.448078 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-dcgm-https-18e37510356f4af5/dcgm-https-exporter" event={"ID":"010ef2cc-5f12-48eb-a708-d6418e0da0af","Type":"ContainerDied","Data":"405104e3721140cad624bec9409f0b469adfffe2b10210eb5c9d08244567d334"} Oct 31 00:19:19.448182 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.448132 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-dcgm-https-18e37510356f4af5/dcgm-https-exporter" event={"ID":"010ef2cc-5f12-48eb-a708-d6418e0da0af","Type":"ContainerDied","Data":"6f6fab66ccca022868e0925ba6afc75a2547668cf76d5bbf43fffcd300ed8478"} Oct 31 00:19:19.448182 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.448156 5274 scope.go:122] "RemoveContainer" containerID="405104e3721140cad624bec9409f0b469adfffe2b10210eb5c9d08244567d334" Oct 31 00:19:19.449599 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.449572 5274 generic.go:356] "Generic (PLEG): container finished" podID="cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246" containerID="4a2962994ed2b9ff68cb5a02374bffa77da04dd75bef522329246d36a5891af4" exitCode=0 Oct 31 00:19:19.449722 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.449602 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246","Type":"ContainerDied","Data":"4a2962994ed2b9ff68cb5a02374bffa77da04dd75bef522329246d36a5891af4"} Oct 31 00:19:19.449722 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.449633 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:19.449722 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.449653 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246","Type":"ContainerDied","Data":"b1ec6e439f7f7807eb8feddfccc7b14ac5eeb8ca52b934dccfaa919ec7ef757d"} Oct 31 00:19:19.457894 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.457872 5274 scope.go:122] "RemoveContainer" containerID="405104e3721140cad624bec9409f0b469adfffe2b10210eb5c9d08244567d334" Oct 31 00:19:19.458243 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:19:19.458212 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"405104e3721140cad624bec9409f0b469adfffe2b10210eb5c9d08244567d334\": container with ID starting with 405104e3721140cad624bec9409f0b469adfffe2b10210eb5c9d08244567d334 not found: ID does not exist" containerID="405104e3721140cad624bec9409f0b469adfffe2b10210eb5c9d08244567d334" Oct 31 00:19:19.458346 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.458249 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"405104e3721140cad624bec9409f0b469adfffe2b10210eb5c9d08244567d334"} err="failed to get container status \"405104e3721140cad624bec9409f0b469adfffe2b10210eb5c9d08244567d334\": rpc error: code = NotFound desc = could not find container \"405104e3721140cad624bec9409f0b469adfffe2b10210eb5c9d08244567d334\": container with ID starting with 405104e3721140cad624bec9409f0b469adfffe2b10210eb5c9d08244567d334 not found: ID does not exist" Oct 31 00:19:19.458346 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.458264 5274 scope.go:122] "RemoveContainer" containerID="4a2962994ed2b9ff68cb5a02374bffa77da04dd75bef522329246d36a5891af4" Oct 31 00:19:19.466548 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.466531 5274 scope.go:122] "RemoveContainer" containerID="4a2962994ed2b9ff68cb5a02374bffa77da04dd75bef522329246d36a5891af4" Oct 31 00:19:19.466829 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:19:19.466810 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a2962994ed2b9ff68cb5a02374bffa77da04dd75bef522329246d36a5891af4\": container with ID starting with 4a2962994ed2b9ff68cb5a02374bffa77da04dd75bef522329246d36a5891af4 not found: ID does not exist" containerID="4a2962994ed2b9ff68cb5a02374bffa77da04dd75bef522329246d36a5891af4" Oct 31 00:19:19.466874 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.466838 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a2962994ed2b9ff68cb5a02374bffa77da04dd75bef522329246d36a5891af4"} err="failed to get container status \"4a2962994ed2b9ff68cb5a02374bffa77da04dd75bef522329246d36a5891af4\": rpc error: code = NotFound desc = could not find container \"4a2962994ed2b9ff68cb5a02374bffa77da04dd75bef522329246d36a5891af4\": container with ID starting with 4a2962994ed2b9ff68cb5a02374bffa77da04dd75bef522329246d36a5891af4 not found: ID does not exist" Oct 31 00:19:19.481219 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.481187 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["tests-monitoring-dcgm-https-18e37510356f4af5/dcgm-https-exporter"] Oct 31 00:19:19.481337 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.481224 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["tests-monitoring-dcgm-https-18e37510356f4af5/dcgm-https-exporter"] Oct 31 00:19:19.494523 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.494501 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:19:19.498463 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.498440 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:19:19.548688 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.548658 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:19:19.549023 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.549011 5274 cpu_manager.go:414] "containerMap: removing container" podUID="cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246" containerName="otc-container" Oct 31 00:19:19.549076 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.549024 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246" containerName="otc-container" Oct 31 00:19:19.549076 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.549052 5274 cpu_manager.go:414] "containerMap: removing container" podUID="010ef2cc-5f12-48eb-a708-d6418e0da0af" containerName="dcgm-https-exporter" Oct 31 00:19:19.549076 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.549057 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="010ef2cc-5f12-48eb-a708-d6418e0da0af" containerName="dcgm-https-exporter" Oct 31 00:19:19.549170 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.549110 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="010ef2cc-5f12-48eb-a708-d6418e0da0af" containerName="dcgm-https-exporter" Oct 31 00:19:19.549170 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.549120 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246" containerName="otc-container" Oct 31 00:19:19.553445 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.553427 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:19.562154 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.562131 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:19:19.562258 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.562173 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:19:19.562258 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.562176 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:19:19.562678 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.562660 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:19:19.562713 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.562692 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-g8x7h\"" Oct 31 00:19:19.562713 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.562706 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:19:19.570701 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.570678 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:19:19.735419 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.735318 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"6ad4c2a9-0c91-4375-b439-f22a29eb45e4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:19.735419 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.735367 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6ad4c2a9-0c91-4375-b439-f22a29eb45e4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:19.735906 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.735499 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mxps\" (UniqueName: \"kubernetes.io/projected/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-kube-api-access-2mxps\") pod \"data-science-collector-collector-0\" (UID: \"6ad4c2a9-0c91-4375-b439-f22a29eb45e4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:19.735906 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.735586 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6ad4c2a9-0c91-4375-b439-f22a29eb45e4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:19.735906 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.735629 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"6ad4c2a9-0c91-4375-b439-f22a29eb45e4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:19.735906 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.735648 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6ad4c2a9-0c91-4375-b439-f22a29eb45e4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:19.836392 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.836353 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6ad4c2a9-0c91-4375-b439-f22a29eb45e4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:19.836571 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.836414 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mxps\" (UniqueName: \"kubernetes.io/projected/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-kube-api-access-2mxps\") pod \"data-science-collector-collector-0\" (UID: \"6ad4c2a9-0c91-4375-b439-f22a29eb45e4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:19.836571 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.836445 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6ad4c2a9-0c91-4375-b439-f22a29eb45e4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:19.836571 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.836475 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"6ad4c2a9-0c91-4375-b439-f22a29eb45e4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:19.836571 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.836490 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6ad4c2a9-0c91-4375-b439-f22a29eb45e4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:19.836571 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.836522 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"6ad4c2a9-0c91-4375-b439-f22a29eb45e4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:19.837347 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.837320 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"6ad4c2a9-0c91-4375-b439-f22a29eb45e4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:19.839007 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.838974 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6ad4c2a9-0c91-4375-b439-f22a29eb45e4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:19.839125 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.839102 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6ad4c2a9-0c91-4375-b439-f22a29eb45e4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:19.839383 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.839368 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"6ad4c2a9-0c91-4375-b439-f22a29eb45e4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:19.839492 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.839472 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6ad4c2a9-0c91-4375-b439-f22a29eb45e4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:19.850002 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.849971 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mxps\" (UniqueName: \"kubernetes.io/projected/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-kube-api-access-2mxps\") pod \"data-science-collector-collector-0\" (UID: \"6ad4c2a9-0c91-4375-b439-f22a29eb45e4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:19.866625 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:19.866594 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:20.144347 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:20.143778 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:19:20.146821 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:19:20.146786 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ad4c2a9_0c91_4375_b439_f22a29eb45e4.slice/crio-dc919f9bfe025cf1bbc237ffa36158b8d4c13967c499f4bd0c50442991adf202 WatchSource:0}: Error finding container dc919f9bfe025cf1bbc237ffa36158b8d4c13967c499f4bd0c50442991adf202: Status 404 returned error can't find the container with id dc919f9bfe025cf1bbc237ffa36158b8d4c13967c499f4bd0c50442991adf202 Oct 31 00:19:20.149021 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:20.148988 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:19:20.149107 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:20.149066 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:19:20.460751 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:20.460657 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"6ad4c2a9-0c91-4375-b439-f22a29eb45e4","Type":"ContainerStarted","Data":"cce7fa950618422cf80ed12d89ec2449c0ed6cd4fccdd91804e28cf49f33d7ae"} Oct 31 00:19:20.460751 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:20.460718 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"6ad4c2a9-0c91-4375-b439-f22a29eb45e4","Type":"ContainerStarted","Data":"dc919f9bfe025cf1bbc237ffa36158b8d4c13967c499f4bd0c50442991adf202"} Oct 31 00:19:20.483559 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:20.483505 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.483491643 podStartE2EDuration="1.483491643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:20.481428976 +0000 UTC m=+1197.118247697" watchObservedRunningTime="2026-10-31 00:19:20.483491643 +0000 UTC m=+1197.120310358" Oct 31 00:19:20.899371 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:19:20.899336 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:19:21.480773 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:21.480740 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="010ef2cc-5f12-48eb-a708-d6418e0da0af" path="/var/lib/kubelet/pods/010ef2cc-5f12-48eb-a708-d6418e0da0af/volumes" Oct 31 00:19:21.481264 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:21.481249 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246" path="/var/lib/kubelet/pods/cb8ea2b6-ad7c-4646-8a2b-ceda7bcaa246/volumes" Oct 31 00:19:36.463412 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:36.463380 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-z67k6"] Oct 31 00:19:36.468797 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:36.468774 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-z67k6" Oct 31 00:19:36.476235 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:36.476208 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-z67k6"] Oct 31 00:19:36.476460 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:36.476241 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"tests-monitoring-tenant-metrics\"/\"openshift-service-ca.crt\"" Oct 31 00:19:36.476460 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:36.476304 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"tests-monitoring-tenant-metrics\"/\"default-dockercfg-qxc8p\"" Oct 31 00:19:36.476460 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:36.476353 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"tests-monitoring-tenant-metrics\"/\"kube-root-ca.crt\"" Oct 31 00:19:36.566306 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:36.566271 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6d369526-5611-433f-8cea-5b7d2e5dd4e9-data\") pod \"tenant-metrics-exporter-5cf544fb97-z67k6\" (UID: \"6d369526-5611-433f-8cea-5b7d2e5dd4e9\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-z67k6" Oct 31 00:19:36.566491 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:36.566334 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wfqt\" (UniqueName: \"kubernetes.io/projected/6d369526-5611-433f-8cea-5b7d2e5dd4e9-kube-api-access-5wfqt\") pod \"tenant-metrics-exporter-5cf544fb97-z67k6\" (UID: \"6d369526-5611-433f-8cea-5b7d2e5dd4e9\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-z67k6" Oct 31 00:19:36.667310 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:36.667276 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wfqt\" (UniqueName: \"kubernetes.io/projected/6d369526-5611-433f-8cea-5b7d2e5dd4e9-kube-api-access-5wfqt\") pod \"tenant-metrics-exporter-5cf544fb97-z67k6\" (UID: \"6d369526-5611-433f-8cea-5b7d2e5dd4e9\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-z67k6" Oct 31 00:19:36.667482 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:36.667384 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6d369526-5611-433f-8cea-5b7d2e5dd4e9-data\") pod \"tenant-metrics-exporter-5cf544fb97-z67k6\" (UID: \"6d369526-5611-433f-8cea-5b7d2e5dd4e9\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-z67k6" Oct 31 00:19:36.667778 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:36.667760 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6d369526-5611-433f-8cea-5b7d2e5dd4e9-data\") pod \"tenant-metrics-exporter-5cf544fb97-z67k6\" (UID: \"6d369526-5611-433f-8cea-5b7d2e5dd4e9\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-z67k6" Oct 31 00:19:36.684709 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:36.684679 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wfqt\" (UniqueName: \"kubernetes.io/projected/6d369526-5611-433f-8cea-5b7d2e5dd4e9-kube-api-access-5wfqt\") pod \"tenant-metrics-exporter-5cf544fb97-z67k6\" (UID: \"6d369526-5611-433f-8cea-5b7d2e5dd4e9\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-z67k6" Oct 31 00:19:36.784459 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:36.784377 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-z67k6" Oct 31 00:19:37.063876 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:37.063848 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-z67k6"] Oct 31 00:19:37.066192 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:19:37.066162 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d369526_5611_433f_8cea_5b7d2e5dd4e9.slice/crio-50f9df18c2024c0a2277a023aaac157a19c0e5446d8a452bf9f7172cc4c5de70 WatchSource:0}: Error finding container 50f9df18c2024c0a2277a023aaac157a19c0e5446d8a452bf9f7172cc4c5de70: Status 404 returned error can't find the container with id 50f9df18c2024c0a2277a023aaac157a19c0e5446d8a452bf9f7172cc4c5de70 Oct 31 00:19:37.551041 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:37.550994 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-z67k6" event={"ID":"6d369526-5611-433f-8cea-5b7d2e5dd4e9","Type":"ContainerStarted","Data":"34b00981964f57fe9744135e7a77103fc06a48a3b7da7b66fb2dec56fe999389"} Oct 31 00:19:37.551529 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:37.551065 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-z67k6" event={"ID":"6d369526-5611-433f-8cea-5b7d2e5dd4e9","Type":"ContainerStarted","Data":"50f9df18c2024c0a2277a023aaac157a19c0e5446d8a452bf9f7172cc4c5de70"} Oct 31 00:19:37.569094 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:37.569035 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-z67k6" podStartSLOduration=1.569020606 podStartE2EDuration="1.569020606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:37.568165506 +0000 UTC m=+1214.204984223" watchObservedRunningTime="2026-10-31 00:19:37.569020606 +0000 UTC m=+1214.205839322" Oct 31 00:19:38.408155 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:38.399543 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:19:38.408155 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:38.399878 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="6ad4c2a9-0c91-4375-b439-f22a29eb45e4" containerName="otc-container" containerID="cri-o://cce7fa950618422cf80ed12d89ec2449c0ed6cd4fccdd91804e28cf49f33d7ae" gracePeriod=30 Oct 31 00:19:40.753469 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:40.753446 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:40.807885 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:40.807857 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-data-science-collector-ta-client-cert\") pod \"6ad4c2a9-0c91-4375-b439-f22a29eb45e4\" (UID: \"6ad4c2a9-0c91-4375-b439-f22a29eb45e4\") " Oct 31 00:19:40.807885 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:40.807889 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-monitoring-tls-certs\") pod \"6ad4c2a9-0c91-4375-b439-f22a29eb45e4\" (UID: \"6ad4c2a9-0c91-4375-b439-f22a29eb45e4\") " Oct 31 00:19:40.808105 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:40.807929 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-kube-api-access-2mxps\" (UniqueName: \"kubernetes.io/projected/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-kube-api-access-2mxps\") pod \"6ad4c2a9-0c91-4375-b439-f22a29eb45e4\" (UID: \"6ad4c2a9-0c91-4375-b439-f22a29eb45e4\") " Oct 31 00:19:40.808105 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:40.807980 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-tls-certs\") pod \"6ad4c2a9-0c91-4375-b439-f22a29eb45e4\" (UID: \"6ad4c2a9-0c91-4375-b439-f22a29eb45e4\") " Oct 31 00:19:40.808105 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:40.808029 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-otc-internal\" (UniqueName: \"kubernetes.io/configmap/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-otc-internal\") pod \"6ad4c2a9-0c91-4375-b439-f22a29eb45e4\" (UID: \"6ad4c2a9-0c91-4375-b439-f22a29eb45e4\") " Oct 31 00:19:40.808105 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:40.808077 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-otlp-tls-certs\") pod \"6ad4c2a9-0c91-4375-b439-f22a29eb45e4\" (UID: \"6ad4c2a9-0c91-4375-b439-f22a29eb45e4\") " Oct 31 00:19:40.808516 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:40.808479 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-otc-internal" pod "6ad4c2a9-0c91-4375-b439-f22a29eb45e4" (UID: "6ad4c2a9-0c91-4375-b439-f22a29eb45e4"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:19:40.808730 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:40.808709 5274 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-otc-internal\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:40.810208 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:40.810177 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-kube-api-access-2mxps" pod "6ad4c2a9-0c91-4375-b439-f22a29eb45e4" (UID: "6ad4c2a9-0c91-4375-b439-f22a29eb45e4"). InnerVolumeSpecName "kube-api-access-2mxps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:19:40.810208 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:40.810191 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-tls-certs" pod "6ad4c2a9-0c91-4375-b439-f22a29eb45e4" (UID: "6ad4c2a9-0c91-4375-b439-f22a29eb45e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:40.810341 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:40.810253 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-data-science-collector-ta-client-cert" pod "6ad4c2a9-0c91-4375-b439-f22a29eb45e4" (UID: "6ad4c2a9-0c91-4375-b439-f22a29eb45e4"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:40.810475 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:40.810456 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-monitoring-tls-certs" pod "6ad4c2a9-0c91-4375-b439-f22a29eb45e4" (UID: "6ad4c2a9-0c91-4375-b439-f22a29eb45e4"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:40.810564 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:40.810474 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-otlp-tls-certs" pod "6ad4c2a9-0c91-4375-b439-f22a29eb45e4" (UID: "6ad4c2a9-0c91-4375-b439-f22a29eb45e4"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:40.909323 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:40.909297 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mxps\" (UniqueName: \"kubernetes.io/projected/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-kube-api-access-2mxps\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:40.909323 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:40.909320 5274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:40.909486 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:40.909329 5274 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-otlp-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:40.909486 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:40.909340 5274 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-data-science-collector-ta-client-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:40.909486 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:40.909349 5274 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ad4c2a9-0c91-4375-b439-f22a29eb45e4-monitoring-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:41.571454 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:41.571420 5274 generic.go:356] "Generic (PLEG): container finished" podID="6ad4c2a9-0c91-4375-b439-f22a29eb45e4" containerID="cce7fa950618422cf80ed12d89ec2449c0ed6cd4fccdd91804e28cf49f33d7ae" exitCode=0 Oct 31 00:19:41.571640 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:41.571489 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:19:41.571640 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:41.571509 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"6ad4c2a9-0c91-4375-b439-f22a29eb45e4","Type":"ContainerDied","Data":"cce7fa950618422cf80ed12d89ec2449c0ed6cd4fccdd91804e28cf49f33d7ae"} Oct 31 00:19:41.571640 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:41.571574 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"6ad4c2a9-0c91-4375-b439-f22a29eb45e4","Type":"ContainerDied","Data":"dc919f9bfe025cf1bbc237ffa36158b8d4c13967c499f4bd0c50442991adf202"} Oct 31 00:19:41.571640 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:41.571603 5274 scope.go:122] "RemoveContainer" containerID="cce7fa950618422cf80ed12d89ec2449c0ed6cd4fccdd91804e28cf49f33d7ae" Oct 31 00:19:41.581237 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:41.581049 5274 scope.go:122] "RemoveContainer" containerID="cce7fa950618422cf80ed12d89ec2449c0ed6cd4fccdd91804e28cf49f33d7ae" Oct 31 00:19:41.581410 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:19:41.581391 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cce7fa950618422cf80ed12d89ec2449c0ed6cd4fccdd91804e28cf49f33d7ae\": container with ID starting with cce7fa950618422cf80ed12d89ec2449c0ed6cd4fccdd91804e28cf49f33d7ae not found: ID does not exist" containerID="cce7fa950618422cf80ed12d89ec2449c0ed6cd4fccdd91804e28cf49f33d7ae" Oct 31 00:19:41.581480 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:41.581422 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce7fa950618422cf80ed12d89ec2449c0ed6cd4fccdd91804e28cf49f33d7ae"} err="failed to get container status \"cce7fa950618422cf80ed12d89ec2449c0ed6cd4fccdd91804e28cf49f33d7ae\": rpc error: code = NotFound desc = could not find container \"cce7fa950618422cf80ed12d89ec2449c0ed6cd4fccdd91804e28cf49f33d7ae\": container with ID starting with cce7fa950618422cf80ed12d89ec2449c0ed6cd4fccdd91804e28cf49f33d7ae not found: ID does not exist" Oct 31 00:19:41.595239 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:41.595207 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:19:41.597547 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:41.597527 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:19:43.481429 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:19:43.481398 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ad4c2a9-0c91-4375-b439-f22a29eb45e4" path="/var/lib/kubelet/pods/6ad4c2a9-0c91-4375-b439-f22a29eb45e4/volumes" Oct 31 00:19:50.900193 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:19:50.900142 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:20:20.898564 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:20:20.898522 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:20:50.898729 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:20:50.898691 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:21:20.898935 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:21:20.898900 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:21:50.899393 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:21:50.899358 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:22:20.898690 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:22:20.898604 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:22:50.899986 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:22:50.899937 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:23:20.899493 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:23:20.899454 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:23:33.212118 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:23:33.212076 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-z67k6"] Oct 31 00:23:33.212525 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:23:33.212278 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-z67k6" podUID="6d369526-5611-433f-8cea-5b7d2e5dd4e9" containerName="tenant-metrics-exporter" containerID="cri-o://34b00981964f57fe9744135e7a77103fc06a48a3b7da7b66fb2dec56fe999389" gracePeriod=30 Oct 31 00:23:33.563631 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:23:33.563609 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-z67k6" Oct 31 00:23:33.671929 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:23:33.671898 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d369526-5611-433f-8cea-5b7d2e5dd4e9-data\" (UniqueName: \"kubernetes.io/empty-dir/6d369526-5611-433f-8cea-5b7d2e5dd4e9-data\") pod \"6d369526-5611-433f-8cea-5b7d2e5dd4e9\" (UID: \"6d369526-5611-433f-8cea-5b7d2e5dd4e9\") " Oct 31 00:23:33.672088 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:23:33.672061 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d369526-5611-433f-8cea-5b7d2e5dd4e9-kube-api-access-5wfqt\" (UniqueName: \"kubernetes.io/projected/6d369526-5611-433f-8cea-5b7d2e5dd4e9-kube-api-access-5wfqt\") pod \"6d369526-5611-433f-8cea-5b7d2e5dd4e9\" (UID: \"6d369526-5611-433f-8cea-5b7d2e5dd4e9\") " Oct 31 00:23:33.672366 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:23:33.672341 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d369526-5611-433f-8cea-5b7d2e5dd4e9-data" pod "6d369526-5611-433f-8cea-5b7d2e5dd4e9" (UID: "6d369526-5611-433f-8cea-5b7d2e5dd4e9"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:23:33.674148 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:23:33.674122 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d369526-5611-433f-8cea-5b7d2e5dd4e9-kube-api-access-5wfqt" pod "6d369526-5611-433f-8cea-5b7d2e5dd4e9" (UID: "6d369526-5611-433f-8cea-5b7d2e5dd4e9"). InnerVolumeSpecName "kube-api-access-5wfqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:23:33.755543 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:23:33.755469 5274 generic.go:356] "Generic (PLEG): container finished" podID="6d369526-5611-433f-8cea-5b7d2e5dd4e9" containerID="34b00981964f57fe9744135e7a77103fc06a48a3b7da7b66fb2dec56fe999389" exitCode=0 Oct 31 00:23:33.755543 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:23:33.755512 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-z67k6" Oct 31 00:23:33.755708 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:23:33.755552 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-z67k6" event={"ID":"6d369526-5611-433f-8cea-5b7d2e5dd4e9","Type":"ContainerDied","Data":"34b00981964f57fe9744135e7a77103fc06a48a3b7da7b66fb2dec56fe999389"} Oct 31 00:23:33.755708 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:23:33.755606 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-z67k6" event={"ID":"6d369526-5611-433f-8cea-5b7d2e5dd4e9","Type":"ContainerDied","Data":"50f9df18c2024c0a2277a023aaac157a19c0e5446d8a452bf9f7172cc4c5de70"} Oct 31 00:23:33.755708 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:23:33.755631 5274 scope.go:122] "RemoveContainer" containerID="34b00981964f57fe9744135e7a77103fc06a48a3b7da7b66fb2dec56fe999389" Oct 31 00:23:33.765228 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:23:33.765211 5274 scope.go:122] "RemoveContainer" containerID="34b00981964f57fe9744135e7a77103fc06a48a3b7da7b66fb2dec56fe999389" Oct 31 00:23:33.765484 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:23:33.765466 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34b00981964f57fe9744135e7a77103fc06a48a3b7da7b66fb2dec56fe999389\": container with ID starting with 34b00981964f57fe9744135e7a77103fc06a48a3b7da7b66fb2dec56fe999389 not found: ID does not exist" containerID="34b00981964f57fe9744135e7a77103fc06a48a3b7da7b66fb2dec56fe999389" Oct 31 00:23:33.765532 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:23:33.765492 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34b00981964f57fe9744135e7a77103fc06a48a3b7da7b66fb2dec56fe999389"} err="failed to get container status \"34b00981964f57fe9744135e7a77103fc06a48a3b7da7b66fb2dec56fe999389\": rpc error: code = NotFound desc = could not find container \"34b00981964f57fe9744135e7a77103fc06a48a3b7da7b66fb2dec56fe999389\": container with ID starting with 34b00981964f57fe9744135e7a77103fc06a48a3b7da7b66fb2dec56fe999389 not found: ID does not exist" Oct 31 00:23:33.773749 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:23:33.773732 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wfqt\" (UniqueName: \"kubernetes.io/projected/6d369526-5611-433f-8cea-5b7d2e5dd4e9-kube-api-access-5wfqt\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:23:33.773819 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:23:33.773751 5274 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6d369526-5611-433f-8cea-5b7d2e5dd4e9-data\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:23:33.778452 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:23:33.778432 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-z67k6"] Oct 31 00:23:33.783478 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:23:33.783454 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-z67k6"] Oct 31 00:23:35.481690 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:23:35.481657 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d369526-5611-433f-8cea-5b7d2e5dd4e9" path="/var/lib/kubelet/pods/6d369526-5611-433f-8cea-5b7d2e5dd4e9/volumes" Oct 31 00:23:50.898806 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:23:50.898772 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:24:00.443966 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:00.443931 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:24:00.446324 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:00.444289 5274 cpu_manager.go:414] "containerMap: removing container" podUID="6ad4c2a9-0c91-4375-b439-f22a29eb45e4" containerName="otc-container" Oct 31 00:24:00.446324 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:00.444299 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ad4c2a9-0c91-4375-b439-f22a29eb45e4" containerName="otc-container" Oct 31 00:24:00.446324 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:00.444330 5274 cpu_manager.go:414] "containerMap: removing container" podUID="6d369526-5611-433f-8cea-5b7d2e5dd4e9" containerName="tenant-metrics-exporter" Oct 31 00:24:00.446324 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:00.444335 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d369526-5611-433f-8cea-5b7d2e5dd4e9" containerName="tenant-metrics-exporter" Oct 31 00:24:00.446324 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:00.444396 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d369526-5611-433f-8cea-5b7d2e5dd4e9" containerName="tenant-metrics-exporter" Oct 31 00:24:00.446324 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:00.444405 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ad4c2a9-0c91-4375-b439-f22a29eb45e4" containerName="otc-container" Oct 31 00:24:00.447579 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:00.447564 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:24:00.452566 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:00.452525 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:24:00.452686 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:00.452645 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-g8x7h\"" Oct 31 00:24:00.452686 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:00.452657 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:24:00.452686 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:00.452664 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:24:00.452811 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:00.452715 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:24:00.453374 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:00.453356 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:24:00.460592 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:00.460569 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:24:00.596968 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:00.596939 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/17d02fc4-f09b-49a6-8517-36c3abc3ae68-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"17d02fc4-f09b-49a6-8517-36c3abc3ae68\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:24:00.597115 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:00.596984 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17d02fc4-f09b-49a6-8517-36c3abc3ae68-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"17d02fc4-f09b-49a6-8517-36c3abc3ae68\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:24:00.597115 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:00.597079 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/17d02fc4-f09b-49a6-8517-36c3abc3ae68-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"17d02fc4-f09b-49a6-8517-36c3abc3ae68\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:24:00.597212 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:00.597199 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/17d02fc4-f09b-49a6-8517-36c3abc3ae68-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"17d02fc4-f09b-49a6-8517-36c3abc3ae68\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:24:00.597251 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:00.597244 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9nkh\" (UniqueName: \"kubernetes.io/projected/17d02fc4-f09b-49a6-8517-36c3abc3ae68-kube-api-access-n9nkh\") pod \"data-science-collector-collector-0\" (UID: \"17d02fc4-f09b-49a6-8517-36c3abc3ae68\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:24:00.597399 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:00.597381 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/17d02fc4-f09b-49a6-8517-36c3abc3ae68-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"17d02fc4-f09b-49a6-8517-36c3abc3ae68\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:24:00.698257 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:00.698197 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/17d02fc4-f09b-49a6-8517-36c3abc3ae68-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"17d02fc4-f09b-49a6-8517-36c3abc3ae68\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:24:00.698257 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:00.698246 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/17d02fc4-f09b-49a6-8517-36c3abc3ae68-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"17d02fc4-f09b-49a6-8517-36c3abc3ae68\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:24:00.698399 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:00.698262 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17d02fc4-f09b-49a6-8517-36c3abc3ae68-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"17d02fc4-f09b-49a6-8517-36c3abc3ae68\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:24:00.698399 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:00.698288 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/17d02fc4-f09b-49a6-8517-36c3abc3ae68-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"17d02fc4-f09b-49a6-8517-36c3abc3ae68\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:24:00.698399 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:00.698329 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/17d02fc4-f09b-49a6-8517-36c3abc3ae68-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"17d02fc4-f09b-49a6-8517-36c3abc3ae68\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:24:00.698399 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:00.698364 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9nkh\" (UniqueName: \"kubernetes.io/projected/17d02fc4-f09b-49a6-8517-36c3abc3ae68-kube-api-access-n9nkh\") pod \"data-science-collector-collector-0\" (UID: \"17d02fc4-f09b-49a6-8517-36c3abc3ae68\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:24:00.698925 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:00.698899 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/17d02fc4-f09b-49a6-8517-36c3abc3ae68-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"17d02fc4-f09b-49a6-8517-36c3abc3ae68\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:24:00.700779 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:00.700753 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/17d02fc4-f09b-49a6-8517-36c3abc3ae68-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"17d02fc4-f09b-49a6-8517-36c3abc3ae68\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:24:00.700890 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:00.700818 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/17d02fc4-f09b-49a6-8517-36c3abc3ae68-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"17d02fc4-f09b-49a6-8517-36c3abc3ae68\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:24:00.700890 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:00.700819 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/17d02fc4-f09b-49a6-8517-36c3abc3ae68-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"17d02fc4-f09b-49a6-8517-36c3abc3ae68\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:24:00.701039 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:00.700892 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17d02fc4-f09b-49a6-8517-36c3abc3ae68-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"17d02fc4-f09b-49a6-8517-36c3abc3ae68\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:24:00.711236 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:00.711214 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9nkh\" (UniqueName: \"kubernetes.io/projected/17d02fc4-f09b-49a6-8517-36c3abc3ae68-kube-api-access-n9nkh\") pod \"data-science-collector-collector-0\" (UID: \"17d02fc4-f09b-49a6-8517-36c3abc3ae68\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:24:00.761098 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:00.761078 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:24:01.035260 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:01.035235 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:24:01.037357 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:24:01.037327 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17d02fc4_f09b_49a6_8517_36c3abc3ae68.slice/crio-68130b69d631279298edb191a71e92a7003e218bb078ffd4f057b4c0276cdd51 WatchSource:0}: Error finding container 68130b69d631279298edb191a71e92a7003e218bb078ffd4f057b4c0276cdd51: Status 404 returned error can't find the container with id 68130b69d631279298edb191a71e92a7003e218bb078ffd4f057b4c0276cdd51 Oct 31 00:24:01.039445 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:01.039414 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:24:01.039544 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:01.039484 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:24:01.902566 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:01.902528 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"17d02fc4-f09b-49a6-8517-36c3abc3ae68","Type":"ContainerStarted","Data":"b675229310452f9ad40962546ac6bf1a78e6463b3594d50d2e90c63e50203df2"} Oct 31 00:24:01.903039 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:01.902586 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"17d02fc4-f09b-49a6-8517-36c3abc3ae68","Type":"ContainerStarted","Data":"68130b69d631279298edb191a71e92a7003e218bb078ffd4f057b4c0276cdd51"} Oct 31 00:24:01.925577 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:24:01.925526 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.9255133309999999 podStartE2EDuration="1.925513331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:24:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:24:01.923206613 +0000 UTC m=+1478.560025329" watchObservedRunningTime="2026-10-31 00:24:01.925513331 +0000 UTC m=+1478.562332047" Oct 31 00:24:20.899796 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:24:20.899760 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:24:50.899098 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:24:50.899057 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:25:20.898883 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:25:20.898803 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:25:50.899526 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:25:50.899483 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:26:20.899749 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:26:20.899713 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:26:50.899599 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:26:50.899556 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:27:20.899097 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:27:20.899059 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:27:50.899209 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:27:50.899177 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:28:20.899456 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:28:20.899422 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:28:41.009485 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.009447 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-lbhb9"] Oct 31 00:28:41.009889 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.009650 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-lbhb9" podUID="b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b" containerName="kube-rbac-proxy" containerID="cri-o://849d4610f9856044f068bd85de6a43ba6737082ac4890701321add9a6c348c78" gracePeriod=30 Oct 31 00:28:41.035391 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.035366 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-snfwk"] Oct 31 00:28:41.035555 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.035536 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-snfwk" podUID="bf3c0b8e-98ee-4551-9f95-2b4290d189a2" containerName="korrel8r" containerID="cri-o://39d4661ab9dd84d3d4855f81237ac32c6993792ddc89705fa22054be2ff29b63" gracePeriod=30 Oct 31 00:28:41.258376 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.258344 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:28:41.258666 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.258628 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6" containerName="alertmanager" containerID="cri-o://91300aef897dfaf7e7d7936ea65f8b0b9f217ff5a63fba306c3e7ef5ae49e799" gracePeriod=120 Oct 31 00:28:41.258666 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.258641 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6" containerName="config-reloader" containerID="cri-o://d8231d74911432ff28005561f0bc6c10dd90d81a1ea1343493ce000faaba01cd" gracePeriod=120 Oct 31 00:28:41.271716 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.271649 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:28:41.271991 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.271949 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="a75a1126-5bff-41dd-a732-5a24e6b4d5e5" containerName="prometheus" containerID="cri-o://0c0cacbbc110c969a184fbd7d90cb2cab919b1678d367df996c58c1f93665e2f" gracePeriod=600 Oct 31 00:28:41.272276 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.272143 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="a75a1126-5bff-41dd-a732-5a24e6b4d5e5" containerName="thanos-sidecar" containerID="cri-o://b8a139d3b41947671950fac51a9ed64f93590f1f2934855c1b9895e63159d0b5" gracePeriod=600 Oct 31 00:28:41.272276 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.272233 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="a75a1126-5bff-41dd-a732-5a24e6b4d5e5" containerName="config-reloader" containerID="cri-o://756ebf069641c84d785c4e79f4a89a1514e483ab40ff6e741113462d4895edb6" gracePeriod=600 Oct 31 00:28:41.317440 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.317409 5274 generic.go:356] "Generic (PLEG): container finished" podID="b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b" containerID="849d4610f9856044f068bd85de6a43ba6737082ac4890701321add9a6c348c78" exitCode=0 Oct 31 00:28:41.317577 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.317483 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-lbhb9" event={"ID":"b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b","Type":"ContainerDied","Data":"849d4610f9856044f068bd85de6a43ba6737082ac4890701321add9a6c348c78"} Oct 31 00:28:41.319455 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.319432 5274 generic.go:356] "Generic (PLEG): container finished" podID="bf3c0b8e-98ee-4551-9f95-2b4290d189a2" containerID="39d4661ab9dd84d3d4855f81237ac32c6993792ddc89705fa22054be2ff29b63" exitCode=0 Oct 31 00:28:41.319661 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.319506 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-snfwk" event={"ID":"bf3c0b8e-98ee-4551-9f95-2b4290d189a2","Type":"ContainerDied","Data":"39d4661ab9dd84d3d4855f81237ac32c6993792ddc89705fa22054be2ff29b63"} Oct 31 00:28:41.403539 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.403511 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-lbhb9" Oct 31 00:28:41.405574 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.405550 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-snfwk" Oct 31 00:28:41.503638 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.503602 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf3c0b8e-98ee-4551-9f95-2b4290d189a2-config\" (UniqueName: \"kubernetes.io/configmap/bf3c0b8e-98ee-4551-9f95-2b4290d189a2-config\") pod \"bf3c0b8e-98ee-4551-9f95-2b4290d189a2\" (UID: \"bf3c0b8e-98ee-4551-9f95-2b4290d189a2\") " Oct 31 00:28:41.503830 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.503672 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf3c0b8e-98ee-4551-9f95-2b4290d189a2-kube-api-access-c59xw\" (UniqueName: \"kubernetes.io/projected/bf3c0b8e-98ee-4551-9f95-2b4290d189a2-kube-api-access-c59xw\") pod \"bf3c0b8e-98ee-4551-9f95-2b4290d189a2\" (UID: \"bf3c0b8e-98ee-4551-9f95-2b4290d189a2\") " Oct 31 00:28:41.503830 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.503753 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b-kube-api-access-pg4rc\" (UniqueName: \"kubernetes.io/projected/b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b-kube-api-access-pg4rc\") pod \"b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b\" (UID: \"b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b\") " Oct 31 00:28:41.503830 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.503796 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b-tls\" (UniqueName: \"kubernetes.io/secret/b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b-tls\") pod \"b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b\" (UID: \"b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b\") " Oct 31 00:28:41.504056 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.503836 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b-prometheus-ca\") pod \"b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b\" (UID: \"b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b\") " Oct 31 00:28:41.504056 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.503877 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf3c0b8e-98ee-4551-9f95-2b4290d189a2-tls\" (UniqueName: \"kubernetes.io/secret/bf3c0b8e-98ee-4551-9f95-2b4290d189a2-tls\") pod \"bf3c0b8e-98ee-4551-9f95-2b4290d189a2\" (UID: \"bf3c0b8e-98ee-4551-9f95-2b4290d189a2\") " Oct 31 00:28:41.504056 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.503916 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b-kube-rbac-proxy-config\") pod \"b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b\" (UID: \"b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b\") " Oct 31 00:28:41.504056 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.503973 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b-prometheus-client-cert\") pod \"b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b\" (UID: \"b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b\") " Oct 31 00:28:41.504302 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.504278 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf3c0b8e-98ee-4551-9f95-2b4290d189a2-config" pod "bf3c0b8e-98ee-4551-9f95-2b4290d189a2" (UID: "bf3c0b8e-98ee-4551-9f95-2b4290d189a2"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:41.504690 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.504670 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b-prometheus-ca" pod "b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b" (UID: "b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:41.506687 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.506661 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:28:41.507218 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.507174 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="17d02fc4-f09b-49a6-8517-36c3abc3ae68" containerName="otc-container" containerID="cri-o://b675229310452f9ad40962546ac6bf1a78e6463b3594d50d2e90c63e50203df2" gracePeriod=30 Oct 31 00:28:41.549704 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.549617 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b-kube-api-access-pg4rc" pod "b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b" (UID: "b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b"). InnerVolumeSpecName "kube-api-access-pg4rc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:41.549794 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.549723 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b-kube-rbac-proxy-config" pod "b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b" (UID: "b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:41.550016 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.549991 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b-prometheus-client-cert" pod "b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b" (UID: "b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:41.550472 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.550453 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf3c0b8e-98ee-4551-9f95-2b4290d189a2-kube-api-access-c59xw" pod "bf3c0b8e-98ee-4551-9f95-2b4290d189a2" (UID: "bf3c0b8e-98ee-4551-9f95-2b4290d189a2"). InnerVolumeSpecName "kube-api-access-c59xw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:41.550597 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.550575 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b-tls" pod "b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b" (UID: "b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:41.550739 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.550709 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf3c0b8e-98ee-4551-9f95-2b4290d189a2-tls" pod "bf3c0b8e-98ee-4551-9f95-2b4290d189a2" (UID: "bf3c0b8e-98ee-4551-9f95-2b4290d189a2"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:41.604888 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.604863 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bf3c0b8e-98ee-4551-9f95-2b4290d189a2-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:41.604888 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.604890 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c59xw\" (UniqueName: \"kubernetes.io/projected/bf3c0b8e-98ee-4551-9f95-2b4290d189a2-kube-api-access-c59xw\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:41.605076 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.604901 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg4rc\" (UniqueName: \"kubernetes.io/projected/b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b-kube-api-access-pg4rc\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:41.605076 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.604910 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:41.605076 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.604921 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b-prometheus-ca\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:41.605076 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.604929 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/bf3c0b8e-98ee-4551-9f95-2b4290d189a2-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:41.605076 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.604937 5274 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b-kube-rbac-proxy-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:41.605076 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.604946 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b-prometheus-client-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:41.784598 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.784566 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:28:41.907698 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.907676 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17d02fc4-f09b-49a6-8517-36c3abc3ae68-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/17d02fc4-f09b-49a6-8517-36c3abc3ae68-monitoring-tls-certs\") pod \"17d02fc4-f09b-49a6-8517-36c3abc3ae68\" (UID: \"17d02fc4-f09b-49a6-8517-36c3abc3ae68\") " Oct 31 00:28:41.907802 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.907722 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17d02fc4-f09b-49a6-8517-36c3abc3ae68-kube-api-access-n9nkh\" (UniqueName: \"kubernetes.io/projected/17d02fc4-f09b-49a6-8517-36c3abc3ae68-kube-api-access-n9nkh\") pod \"17d02fc4-f09b-49a6-8517-36c3abc3ae68\" (UID: \"17d02fc4-f09b-49a6-8517-36c3abc3ae68\") " Oct 31 00:28:41.907802 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.907742 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17d02fc4-f09b-49a6-8517-36c3abc3ae68-tls-certs\" (UniqueName: \"kubernetes.io/secret/17d02fc4-f09b-49a6-8517-36c3abc3ae68-tls-certs\") pod \"17d02fc4-f09b-49a6-8517-36c3abc3ae68\" (UID: \"17d02fc4-f09b-49a6-8517-36c3abc3ae68\") " Oct 31 00:28:41.907920 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.907899 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17d02fc4-f09b-49a6-8517-36c3abc3ae68-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/17d02fc4-f09b-49a6-8517-36c3abc3ae68-data-science-collector-ta-client-cert\") pod \"17d02fc4-f09b-49a6-8517-36c3abc3ae68\" (UID: \"17d02fc4-f09b-49a6-8517-36c3abc3ae68\") " Oct 31 00:28:41.907998 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.907947 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17d02fc4-f09b-49a6-8517-36c3abc3ae68-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/17d02fc4-f09b-49a6-8517-36c3abc3ae68-otlp-tls-certs\") pod \"17d02fc4-f09b-49a6-8517-36c3abc3ae68\" (UID: \"17d02fc4-f09b-49a6-8517-36c3abc3ae68\") " Oct 31 00:28:41.908059 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.908019 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17d02fc4-f09b-49a6-8517-36c3abc3ae68-otc-internal\" (UniqueName: \"kubernetes.io/configmap/17d02fc4-f09b-49a6-8517-36c3abc3ae68-otc-internal\") pod \"17d02fc4-f09b-49a6-8517-36c3abc3ae68\" (UID: \"17d02fc4-f09b-49a6-8517-36c3abc3ae68\") " Oct 31 00:28:41.908673 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.908644 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17d02fc4-f09b-49a6-8517-36c3abc3ae68-otc-internal" pod "17d02fc4-f09b-49a6-8517-36c3abc3ae68" (UID: "17d02fc4-f09b-49a6-8517-36c3abc3ae68"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:41.910096 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.910069 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17d02fc4-f09b-49a6-8517-36c3abc3ae68-kube-api-access-n9nkh" pod "17d02fc4-f09b-49a6-8517-36c3abc3ae68" (UID: "17d02fc4-f09b-49a6-8517-36c3abc3ae68"). InnerVolumeSpecName "kube-api-access-n9nkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:41.910204 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.910134 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17d02fc4-f09b-49a6-8517-36c3abc3ae68-data-science-collector-ta-client-cert" pod "17d02fc4-f09b-49a6-8517-36c3abc3ae68" (UID: "17d02fc4-f09b-49a6-8517-36c3abc3ae68"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:41.910246 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.910213 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17d02fc4-f09b-49a6-8517-36c3abc3ae68-monitoring-tls-certs" pod "17d02fc4-f09b-49a6-8517-36c3abc3ae68" (UID: "17d02fc4-f09b-49a6-8517-36c3abc3ae68"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:41.910342 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.910322 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17d02fc4-f09b-49a6-8517-36c3abc3ae68-tls-certs" pod "17d02fc4-f09b-49a6-8517-36c3abc3ae68" (UID: "17d02fc4-f09b-49a6-8517-36c3abc3ae68"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:41.910381 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:41.910340 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17d02fc4-f09b-49a6-8517-36c3abc3ae68-otlp-tls-certs" pod "17d02fc4-f09b-49a6-8517-36c3abc3ae68" (UID: "17d02fc4-f09b-49a6-8517-36c3abc3ae68"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:42.009620 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.008808 5274 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/17d02fc4-f09b-49a6-8517-36c3abc3ae68-data-science-collector-ta-client-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.009620 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.008841 5274 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/17d02fc4-f09b-49a6-8517-36c3abc3ae68-otlp-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.009620 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.008858 5274 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/17d02fc4-f09b-49a6-8517-36c3abc3ae68-otc-internal\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.009620 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.008872 5274 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/17d02fc4-f09b-49a6-8517-36c3abc3ae68-monitoring-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.009620 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.008887 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9nkh\" (UniqueName: \"kubernetes.io/projected/17d02fc4-f09b-49a6-8517-36c3abc3ae68-kube-api-access-n9nkh\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.009620 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.008903 5274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17d02fc4-f09b-49a6-8517-36c3abc3ae68-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.063930 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.063907 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:28:42.109385 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.109361 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-config-out\" (UniqueName: \"kubernetes.io/empty-dir/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-config-out\") pod \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " Oct 31 00:28:42.109501 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.109397 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " Oct 31 00:28:42.109501 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.109416 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-web-config\" (UniqueName: \"kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-web-config\") pod \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " Oct 31 00:28:42.109501 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.109450 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-tls-assets\" (UniqueName: \"kubernetes.io/projected/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-tls-assets\") pod \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " Oct 31 00:28:42.109664 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.109497 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " Oct 31 00:28:42.109664 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.109536 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-secret-prometheus-web-tls-ca\") pod \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " Oct 31 00:28:42.109664 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.109565 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " Oct 31 00:28:42.109828 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.109665 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " Oct 31 00:28:42.109828 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.109712 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-config\" (UniqueName: \"kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-config\") pod \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " Oct 31 00:28:42.109828 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.109728 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-prometheus-data-science-monitoringstack-rulefiles-2" pod "a75a1126-5bff-41dd-a732-5a24e6b4d5e5" (UID: "a75a1126-5bff-41dd-a732-5a24e6b4d5e5"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:42.109828 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.109757 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-thanos-prometheus-http-client-file\") pod \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " Oct 31 00:28:42.109828 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.109788 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " Oct 31 00:28:42.109828 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.109826 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-kube-api-access-g94lz\" (UniqueName: \"kubernetes.io/projected/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-kube-api-access-g94lz\") pod \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " Oct 31 00:28:42.111698 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.111284 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\" (UID: \"a75a1126-5bff-41dd-a732-5a24e6b4d5e5\") " Oct 31 00:28:42.111698 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.111321 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-prometheus-data-science-monitoringstack-rulefiles-1" pod "a75a1126-5bff-41dd-a732-5a24e6b4d5e5" (UID: "a75a1126-5bff-41dd-a732-5a24e6b4d5e5"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:42.111698 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.111645 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.111698 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.111668 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.112188 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.112152 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-prometheus-data-science-monitoringstack-rulefiles-0" pod "a75a1126-5bff-41dd-a732-5a24e6b4d5e5" (UID: "a75a1126-5bff-41dd-a732-5a24e6b4d5e5"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:42.112726 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.112476 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-kube-api-access-g94lz" pod "a75a1126-5bff-41dd-a732-5a24e6b4d5e5" (UID: "a75a1126-5bff-41dd-a732-5a24e6b4d5e5"). InnerVolumeSpecName "kube-api-access-g94lz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:42.112726 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.112594 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-config-out" pod "a75a1126-5bff-41dd-a732-5a24e6b4d5e5" (UID: "a75a1126-5bff-41dd-a732-5a24e6b4d5e5"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:42.112726 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.112680 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-secret-prometheus-web-tls-ca" pod "a75a1126-5bff-41dd-a732-5a24e6b4d5e5" (UID: "a75a1126-5bff-41dd-a732-5a24e6b4d5e5"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:42.113981 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.113924 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "a75a1126-5bff-41dd-a732-5a24e6b4d5e5" (UID: "a75a1126-5bff-41dd-a732-5a24e6b4d5e5"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:42.114130 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.114105 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-tls-assets" pod "a75a1126-5bff-41dd-a732-5a24e6b4d5e5" (UID: "a75a1126-5bff-41dd-a732-5a24e6b4d5e5"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:42.114227 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.114179 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-thanos-prometheus-http-client-file" pod "a75a1126-5bff-41dd-a732-5a24e6b4d5e5" (UID: "a75a1126-5bff-41dd-a732-5a24e6b4d5e5"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:42.114547 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.114514 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "a75a1126-5bff-41dd-a732-5a24e6b4d5e5" (UID: "a75a1126-5bff-41dd-a732-5a24e6b4d5e5"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:42.114791 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.114768 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-config" pod "a75a1126-5bff-41dd-a732-5a24e6b4d5e5" (UID: "a75a1126-5bff-41dd-a732-5a24e6b4d5e5"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:42.117132 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.117114 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1" pod "a75a1126-5bff-41dd-a732-5a24e6b4d5e5" (UID: "a75a1126-5bff-41dd-a732-5a24e6b4d5e5"). InnerVolumeSpecName "pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:28:42.126321 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.126299 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-web-config" pod "a75a1126-5bff-41dd-a732-5a24e6b4d5e5" (UID: "a75a1126-5bff-41dd-a732-5a24e6b4d5e5"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:42.135887 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.135868 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:28:42.212085 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.212019 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-config-out\" (UniqueName: \"kubernetes.io/empty-dir/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-config-out\") pod \"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6\" (UID: \"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6\") " Oct 31 00:28:42.212085 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.212069 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-alertmanager-data-science-monitoringstack-db\") pod \"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6\" (UID: \"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6\") " Oct 31 00:28:42.212258 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.212105 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-config-volume\" (UniqueName: \"kubernetes.io/secret/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-config-volume\") pod \"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6\" (UID: \"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6\") " Oct 31 00:28:42.212258 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.212137 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-kube-api-access-tt6zz\" (UniqueName: \"kubernetes.io/projected/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-kube-api-access-tt6zz\") pod \"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6\" (UID: \"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6\") " Oct 31 00:28:42.212258 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.212172 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-web-config\" (UniqueName: \"kubernetes.io/secret/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-web-config\") pod \"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6\" (UID: \"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6\") " Oct 31 00:28:42.212258 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.212203 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-cluster-tls-config\") pod \"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6\" (UID: \"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6\") " Oct 31 00:28:42.212258 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.212236 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-tls-assets\" (UniqueName: \"kubernetes.io/projected/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-tls-assets\") pod \"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6\" (UID: \"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6\") " Oct 31 00:28:42.212520 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.212469 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-alertmanager-data-science-monitoringstack-db" pod "72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6" (UID: "72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:42.212520 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.212479 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g94lz\" (UniqueName: \"kubernetes.io/projected/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-kube-api-access-g94lz\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.212634 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.212519 5274 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.212634 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.212538 5274 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-config-out\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.212634 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.212552 5274 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-web-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.212634 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.212567 5274 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-tls-assets\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.212634 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.212581 5274 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.212634 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.212596 5274 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-secret-prometheus-web-tls-ca\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.212634 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.212632 5274 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" " Oct 31 00:28:42.213007 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.212645 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.213007 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.212660 5274 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.213007 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.212675 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a75a1126-5bff-41dd-a732-5a24e6b4d5e5-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.214420 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.214392 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-kube-api-access-tt6zz" pod "72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6" (UID: "72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6"). InnerVolumeSpecName "kube-api-access-tt6zz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:42.214768 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.214736 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-tls-assets" pod "72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6" (UID: "72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:42.214864 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.214812 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-config-out" pod "72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6" (UID: "72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:42.214864 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.214831 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-config-volume" pod "72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6" (UID: "72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:42.219535 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.219512 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-cluster-tls-config" pod "72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6" (UID: "72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:42.225715 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.225688 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-web-config" pod "72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6" (UID: "72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:42.225791 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.225727 5274 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1") on node "ip-10-0-129-70.us-east-2.compute.internal" Oct 31 00:28:42.313279 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.313246 5274 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-web-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.313279 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.313275 5274 reconciler_common.go:299] "Volume detached for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"csi-51e8027f163b5f98040faf3f8c30848dd44a7a90ca2bd0af2446ea6398ed018d\"" Oct 31 00:28:42.313279 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.313285 5274 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-cluster-tls-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.313543 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.313296 5274 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-tls-assets\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.313543 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.313304 5274 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-config-out\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.313543 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.313314 5274 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.313543 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.313323 5274 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-config-volume\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.313543 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.313333 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt6zz\" (UniqueName: \"kubernetes.io/projected/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6-kube-api-access-tt6zz\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.327352 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.327321 5274 generic.go:356] "Generic (PLEG): container finished" podID="72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6" containerID="d8231d74911432ff28005561f0bc6c10dd90d81a1ea1343493ce000faaba01cd" exitCode=0 Oct 31 00:28:42.327352 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.327344 5274 generic.go:356] "Generic (PLEG): container finished" podID="72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6" containerID="91300aef897dfaf7e7d7936ea65f8b0b9f217ff5a63fba306c3e7ef5ae49e799" exitCode=0 Oct 31 00:28:42.327633 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.327377 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:28:42.327633 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.327394 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6","Type":"ContainerDied","Data":"d8231d74911432ff28005561f0bc6c10dd90d81a1ea1343493ce000faaba01cd"} Oct 31 00:28:42.327633 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.327462 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6","Type":"ContainerDied","Data":"91300aef897dfaf7e7d7936ea65f8b0b9f217ff5a63fba306c3e7ef5ae49e799"} Oct 31 00:28:42.327633 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.327496 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6","Type":"ContainerDied","Data":"70296fd8d231724b8706579b20adfa128d8990dfbef5971192987a92425006d4"} Oct 31 00:28:42.327633 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.327539 5274 scope.go:122] "RemoveContainer" containerID="d8231d74911432ff28005561f0bc6c10dd90d81a1ea1343493ce000faaba01cd" Oct 31 00:28:42.328969 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.328934 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-snfwk" Oct 31 00:28:42.329115 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.328986 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-snfwk" event={"ID":"bf3c0b8e-98ee-4551-9f95-2b4290d189a2","Type":"ContainerDied","Data":"5119983d524e5c203de251adc6ecd444135e25071e75a3460c033257207bcf78"} Oct 31 00:28:42.330613 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.330563 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-lbhb9" Oct 31 00:28:42.330613 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.330585 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-lbhb9" event={"ID":"b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b","Type":"ContainerDied","Data":"ed84bae0cbe0b63afef44ea48c407cc14a7ab99a62d28f729e37476dd156eb5d"} Oct 31 00:28:42.333425 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.333402 5274 generic.go:356] "Generic (PLEG): container finished" podID="a75a1126-5bff-41dd-a732-5a24e6b4d5e5" containerID="b8a139d3b41947671950fac51a9ed64f93590f1f2934855c1b9895e63159d0b5" exitCode=0 Oct 31 00:28:42.333576 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.333427 5274 generic.go:356] "Generic (PLEG): container finished" podID="a75a1126-5bff-41dd-a732-5a24e6b4d5e5" containerID="756ebf069641c84d785c4e79f4a89a1514e483ab40ff6e741113462d4895edb6" exitCode=0 Oct 31 00:28:42.333576 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.333437 5274 generic.go:356] "Generic (PLEG): container finished" podID="a75a1126-5bff-41dd-a732-5a24e6b4d5e5" containerID="0c0cacbbc110c969a184fbd7d90cb2cab919b1678d367df996c58c1f93665e2f" exitCode=0 Oct 31 00:28:42.333576 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.333516 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"a75a1126-5bff-41dd-a732-5a24e6b4d5e5","Type":"ContainerDied","Data":"b8a139d3b41947671950fac51a9ed64f93590f1f2934855c1b9895e63159d0b5"} Oct 31 00:28:42.333576 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.333569 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"a75a1126-5bff-41dd-a732-5a24e6b4d5e5","Type":"ContainerDied","Data":"756ebf069641c84d785c4e79f4a89a1514e483ab40ff6e741113462d4895edb6"} Oct 31 00:28:42.333886 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.333606 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"a75a1126-5bff-41dd-a732-5a24e6b4d5e5","Type":"ContainerDied","Data":"0c0cacbbc110c969a184fbd7d90cb2cab919b1678d367df996c58c1f93665e2f"} Oct 31 00:28:42.333886 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.333641 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"a75a1126-5bff-41dd-a732-5a24e6b4d5e5","Type":"ContainerDied","Data":"125bd07a37dc61a8c256b88c6250b98396a9bbde3086f54243ba9fc4437c4b75"} Oct 31 00:28:42.333886 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.333719 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:28:42.342437 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.338928 5274 scope.go:122] "RemoveContainer" containerID="91300aef897dfaf7e7d7936ea65f8b0b9f217ff5a63fba306c3e7ef5ae49e799" Oct 31 00:28:42.342437 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.339750 5274 generic.go:356] "Generic (PLEG): container finished" podID="17d02fc4-f09b-49a6-8517-36c3abc3ae68" containerID="b675229310452f9ad40962546ac6bf1a78e6463b3594d50d2e90c63e50203df2" exitCode=0 Oct 31 00:28:42.342437 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.339781 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"17d02fc4-f09b-49a6-8517-36c3abc3ae68","Type":"ContainerDied","Data":"b675229310452f9ad40962546ac6bf1a78e6463b3594d50d2e90c63e50203df2"} Oct 31 00:28:42.342437 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.339865 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"17d02fc4-f09b-49a6-8517-36c3abc3ae68","Type":"ContainerDied","Data":"68130b69d631279298edb191a71e92a7003e218bb078ffd4f057b4c0276cdd51"} Oct 31 00:28:42.342437 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.340061 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:28:42.352295 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.352274 5274 scope.go:122] "RemoveContainer" containerID="aeb605a44bc2f84f9a3d05f7bc0a8af46fd1f72351fe75d9f2eb966cc5ca63a6" Oct 31 00:28:42.360077 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.360058 5274 scope.go:122] "RemoveContainer" containerID="d8231d74911432ff28005561f0bc6c10dd90d81a1ea1343493ce000faaba01cd" Oct 31 00:28:42.360329 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:28:42.360311 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8231d74911432ff28005561f0bc6c10dd90d81a1ea1343493ce000faaba01cd\": container with ID starting with d8231d74911432ff28005561f0bc6c10dd90d81a1ea1343493ce000faaba01cd not found: ID does not exist" containerID="d8231d74911432ff28005561f0bc6c10dd90d81a1ea1343493ce000faaba01cd" Oct 31 00:28:42.360373 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.360336 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8231d74911432ff28005561f0bc6c10dd90d81a1ea1343493ce000faaba01cd"} err="failed to get container status \"d8231d74911432ff28005561f0bc6c10dd90d81a1ea1343493ce000faaba01cd\": rpc error: code = NotFound desc = could not find container \"d8231d74911432ff28005561f0bc6c10dd90d81a1ea1343493ce000faaba01cd\": container with ID starting with d8231d74911432ff28005561f0bc6c10dd90d81a1ea1343493ce000faaba01cd not found: ID does not exist" Oct 31 00:28:42.360373 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.360351 5274 scope.go:122] "RemoveContainer" containerID="91300aef897dfaf7e7d7936ea65f8b0b9f217ff5a63fba306c3e7ef5ae49e799" Oct 31 00:28:42.360604 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:28:42.360585 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91300aef897dfaf7e7d7936ea65f8b0b9f217ff5a63fba306c3e7ef5ae49e799\": container with ID starting with 91300aef897dfaf7e7d7936ea65f8b0b9f217ff5a63fba306c3e7ef5ae49e799 not found: ID does not exist" containerID="91300aef897dfaf7e7d7936ea65f8b0b9f217ff5a63fba306c3e7ef5ae49e799" Oct 31 00:28:42.360663 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.360609 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91300aef897dfaf7e7d7936ea65f8b0b9f217ff5a63fba306c3e7ef5ae49e799"} err="failed to get container status \"91300aef897dfaf7e7d7936ea65f8b0b9f217ff5a63fba306c3e7ef5ae49e799\": rpc error: code = NotFound desc = could not find container \"91300aef897dfaf7e7d7936ea65f8b0b9f217ff5a63fba306c3e7ef5ae49e799\": container with ID starting with 91300aef897dfaf7e7d7936ea65f8b0b9f217ff5a63fba306c3e7ef5ae49e799 not found: ID does not exist" Oct 31 00:28:42.360663 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.360624 5274 scope.go:122] "RemoveContainer" containerID="aeb605a44bc2f84f9a3d05f7bc0a8af46fd1f72351fe75d9f2eb966cc5ca63a6" Oct 31 00:28:42.360848 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:28:42.360830 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeb605a44bc2f84f9a3d05f7bc0a8af46fd1f72351fe75d9f2eb966cc5ca63a6\": container with ID starting with aeb605a44bc2f84f9a3d05f7bc0a8af46fd1f72351fe75d9f2eb966cc5ca63a6 not found: ID does not exist" containerID="aeb605a44bc2f84f9a3d05f7bc0a8af46fd1f72351fe75d9f2eb966cc5ca63a6" Oct 31 00:28:42.360903 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.360851 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeb605a44bc2f84f9a3d05f7bc0a8af46fd1f72351fe75d9f2eb966cc5ca63a6"} err="failed to get container status \"aeb605a44bc2f84f9a3d05f7bc0a8af46fd1f72351fe75d9f2eb966cc5ca63a6\": rpc error: code = NotFound desc = could not find container \"aeb605a44bc2f84f9a3d05f7bc0a8af46fd1f72351fe75d9f2eb966cc5ca63a6\": container with ID starting with aeb605a44bc2f84f9a3d05f7bc0a8af46fd1f72351fe75d9f2eb966cc5ca63a6 not found: ID does not exist" Oct 31 00:28:42.360903 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.360863 5274 scope.go:122] "RemoveContainer" containerID="d8231d74911432ff28005561f0bc6c10dd90d81a1ea1343493ce000faaba01cd" Oct 31 00:28:42.361352 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.361330 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8231d74911432ff28005561f0bc6c10dd90d81a1ea1343493ce000faaba01cd"} err="failed to get container status \"d8231d74911432ff28005561f0bc6c10dd90d81a1ea1343493ce000faaba01cd\": rpc error: code = NotFound desc = could not find container \"d8231d74911432ff28005561f0bc6c10dd90d81a1ea1343493ce000faaba01cd\": container with ID starting with d8231d74911432ff28005561f0bc6c10dd90d81a1ea1343493ce000faaba01cd not found: ID does not exist" Oct 31 00:28:42.361352 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.361351 5274 scope.go:122] "RemoveContainer" containerID="91300aef897dfaf7e7d7936ea65f8b0b9f217ff5a63fba306c3e7ef5ae49e799" Oct 31 00:28:42.361607 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.361587 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91300aef897dfaf7e7d7936ea65f8b0b9f217ff5a63fba306c3e7ef5ae49e799"} err="failed to get container status \"91300aef897dfaf7e7d7936ea65f8b0b9f217ff5a63fba306c3e7ef5ae49e799\": rpc error: code = NotFound desc = could not find container \"91300aef897dfaf7e7d7936ea65f8b0b9f217ff5a63fba306c3e7ef5ae49e799\": container with ID starting with 91300aef897dfaf7e7d7936ea65f8b0b9f217ff5a63fba306c3e7ef5ae49e799 not found: ID does not exist" Oct 31 00:28:42.361658 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.361609 5274 scope.go:122] "RemoveContainer" containerID="aeb605a44bc2f84f9a3d05f7bc0a8af46fd1f72351fe75d9f2eb966cc5ca63a6" Oct 31 00:28:42.361826 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.361809 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeb605a44bc2f84f9a3d05f7bc0a8af46fd1f72351fe75d9f2eb966cc5ca63a6"} err="failed to get container status \"aeb605a44bc2f84f9a3d05f7bc0a8af46fd1f72351fe75d9f2eb966cc5ca63a6\": rpc error: code = NotFound desc = could not find container \"aeb605a44bc2f84f9a3d05f7bc0a8af46fd1f72351fe75d9f2eb966cc5ca63a6\": container with ID starting with aeb605a44bc2f84f9a3d05f7bc0a8af46fd1f72351fe75d9f2eb966cc5ca63a6 not found: ID does not exist" Oct 31 00:28:42.361871 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.361826 5274 scope.go:122] "RemoveContainer" containerID="39d4661ab9dd84d3d4855f81237ac32c6993792ddc89705fa22054be2ff29b63" Oct 31 00:28:42.368761 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.368744 5274 scope.go:122] "RemoveContainer" containerID="849d4610f9856044f068bd85de6a43ba6737082ac4890701321add9a6c348c78" Oct 31 00:28:42.371710 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.371693 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-snfwk"] Oct 31 00:28:42.377867 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.376584 5274 scope.go:122] "RemoveContainer" containerID="b8a139d3b41947671950fac51a9ed64f93590f1f2934855c1b9895e63159d0b5" Oct 31 00:28:42.378521 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.378503 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-snfwk"] Oct 31 00:28:42.384199 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.384179 5274 scope.go:122] "RemoveContainer" containerID="756ebf069641c84d785c4e79f4a89a1514e483ab40ff6e741113462d4895edb6" Oct 31 00:28:42.391413 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.391390 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:28:42.392036 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.392018 5274 scope.go:122] "RemoveContainer" containerID="0c0cacbbc110c969a184fbd7d90cb2cab919b1678d367df996c58c1f93665e2f" Oct 31 00:28:42.397787 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.397769 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:28:42.400775 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.400760 5274 scope.go:122] "RemoveContainer" containerID="76493e766a972779c8d756016833221c293a42d884ec55d4025e79273ee7dbde" Oct 31 00:28:42.409227 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.409198 5274 scope.go:122] "RemoveContainer" containerID="b8a139d3b41947671950fac51a9ed64f93590f1f2934855c1b9895e63159d0b5" Oct 31 00:28:42.409439 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:28:42.409424 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8a139d3b41947671950fac51a9ed64f93590f1f2934855c1b9895e63159d0b5\": container with ID starting with b8a139d3b41947671950fac51a9ed64f93590f1f2934855c1b9895e63159d0b5 not found: ID does not exist" containerID="b8a139d3b41947671950fac51a9ed64f93590f1f2934855c1b9895e63159d0b5" Oct 31 00:28:42.409492 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.409447 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8a139d3b41947671950fac51a9ed64f93590f1f2934855c1b9895e63159d0b5"} err="failed to get container status \"b8a139d3b41947671950fac51a9ed64f93590f1f2934855c1b9895e63159d0b5\": rpc error: code = NotFound desc = could not find container \"b8a139d3b41947671950fac51a9ed64f93590f1f2934855c1b9895e63159d0b5\": container with ID starting with b8a139d3b41947671950fac51a9ed64f93590f1f2934855c1b9895e63159d0b5 not found: ID does not exist" Oct 31 00:28:42.409492 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.409465 5274 scope.go:122] "RemoveContainer" containerID="756ebf069641c84d785c4e79f4a89a1514e483ab40ff6e741113462d4895edb6" Oct 31 00:28:42.409732 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:28:42.409716 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"756ebf069641c84d785c4e79f4a89a1514e483ab40ff6e741113462d4895edb6\": container with ID starting with 756ebf069641c84d785c4e79f4a89a1514e483ab40ff6e741113462d4895edb6 not found: ID does not exist" containerID="756ebf069641c84d785c4e79f4a89a1514e483ab40ff6e741113462d4895edb6" Oct 31 00:28:42.409792 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.409738 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"756ebf069641c84d785c4e79f4a89a1514e483ab40ff6e741113462d4895edb6"} err="failed to get container status \"756ebf069641c84d785c4e79f4a89a1514e483ab40ff6e741113462d4895edb6\": rpc error: code = NotFound desc = could not find container \"756ebf069641c84d785c4e79f4a89a1514e483ab40ff6e741113462d4895edb6\": container with ID starting with 756ebf069641c84d785c4e79f4a89a1514e483ab40ff6e741113462d4895edb6 not found: ID does not exist" Oct 31 00:28:42.409792 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.409751 5274 scope.go:122] "RemoveContainer" containerID="0c0cacbbc110c969a184fbd7d90cb2cab919b1678d367df996c58c1f93665e2f" Oct 31 00:28:42.410038 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:28:42.410021 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c0cacbbc110c969a184fbd7d90cb2cab919b1678d367df996c58c1f93665e2f\": container with ID starting with 0c0cacbbc110c969a184fbd7d90cb2cab919b1678d367df996c58c1f93665e2f not found: ID does not exist" containerID="0c0cacbbc110c969a184fbd7d90cb2cab919b1678d367df996c58c1f93665e2f" Oct 31 00:28:42.410090 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.410043 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c0cacbbc110c969a184fbd7d90cb2cab919b1678d367df996c58c1f93665e2f"} err="failed to get container status \"0c0cacbbc110c969a184fbd7d90cb2cab919b1678d367df996c58c1f93665e2f\": rpc error: code = NotFound desc = could not find container \"0c0cacbbc110c969a184fbd7d90cb2cab919b1678d367df996c58c1f93665e2f\": container with ID starting with 0c0cacbbc110c969a184fbd7d90cb2cab919b1678d367df996c58c1f93665e2f not found: ID does not exist" Oct 31 00:28:42.410090 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.410062 5274 scope.go:122] "RemoveContainer" containerID="76493e766a972779c8d756016833221c293a42d884ec55d4025e79273ee7dbde" Oct 31 00:28:42.410245 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.410230 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:28:42.410286 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:28:42.410269 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76493e766a972779c8d756016833221c293a42d884ec55d4025e79273ee7dbde\": container with ID starting with 76493e766a972779c8d756016833221c293a42d884ec55d4025e79273ee7dbde not found: ID does not exist" containerID="76493e766a972779c8d756016833221c293a42d884ec55d4025e79273ee7dbde" Oct 31 00:28:42.410322 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.410293 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76493e766a972779c8d756016833221c293a42d884ec55d4025e79273ee7dbde"} err="failed to get container status \"76493e766a972779c8d756016833221c293a42d884ec55d4025e79273ee7dbde\": rpc error: code = NotFound desc = could not find container \"76493e766a972779c8d756016833221c293a42d884ec55d4025e79273ee7dbde\": container with ID starting with 76493e766a972779c8d756016833221c293a42d884ec55d4025e79273ee7dbde not found: ID does not exist" Oct 31 00:28:42.410322 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.410304 5274 scope.go:122] "RemoveContainer" containerID="b8a139d3b41947671950fac51a9ed64f93590f1f2934855c1b9895e63159d0b5" Oct 31 00:28:42.410512 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.410496 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8a139d3b41947671950fac51a9ed64f93590f1f2934855c1b9895e63159d0b5"} err="failed to get container status \"b8a139d3b41947671950fac51a9ed64f93590f1f2934855c1b9895e63159d0b5\": rpc error: code = NotFound desc = could not find container \"b8a139d3b41947671950fac51a9ed64f93590f1f2934855c1b9895e63159d0b5\": container with ID starting with b8a139d3b41947671950fac51a9ed64f93590f1f2934855c1b9895e63159d0b5 not found: ID does not exist" Oct 31 00:28:42.410555 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.410512 5274 scope.go:122] "RemoveContainer" containerID="756ebf069641c84d785c4e79f4a89a1514e483ab40ff6e741113462d4895edb6" Oct 31 00:28:42.410720 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.410705 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"756ebf069641c84d785c4e79f4a89a1514e483ab40ff6e741113462d4895edb6"} err="failed to get container status \"756ebf069641c84d785c4e79f4a89a1514e483ab40ff6e741113462d4895edb6\": rpc error: code = NotFound desc = could not find container \"756ebf069641c84d785c4e79f4a89a1514e483ab40ff6e741113462d4895edb6\": container with ID starting with 756ebf069641c84d785c4e79f4a89a1514e483ab40ff6e741113462d4895edb6 not found: ID does not exist" Oct 31 00:28:42.410720 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.410720 5274 scope.go:122] "RemoveContainer" containerID="0c0cacbbc110c969a184fbd7d90cb2cab919b1678d367df996c58c1f93665e2f" Oct 31 00:28:42.410991 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.410970 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c0cacbbc110c969a184fbd7d90cb2cab919b1678d367df996c58c1f93665e2f"} err="failed to get container status \"0c0cacbbc110c969a184fbd7d90cb2cab919b1678d367df996c58c1f93665e2f\": rpc error: code = NotFound desc = could not find container \"0c0cacbbc110c969a184fbd7d90cb2cab919b1678d367df996c58c1f93665e2f\": container with ID starting with 0c0cacbbc110c969a184fbd7d90cb2cab919b1678d367df996c58c1f93665e2f not found: ID does not exist" Oct 31 00:28:42.411054 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.410992 5274 scope.go:122] "RemoveContainer" containerID="76493e766a972779c8d756016833221c293a42d884ec55d4025e79273ee7dbde" Oct 31 00:28:42.411191 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.411175 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76493e766a972779c8d756016833221c293a42d884ec55d4025e79273ee7dbde"} err="failed to get container status \"76493e766a972779c8d756016833221c293a42d884ec55d4025e79273ee7dbde\": rpc error: code = NotFound desc = could not find container \"76493e766a972779c8d756016833221c293a42d884ec55d4025e79273ee7dbde\": container with ID starting with 76493e766a972779c8d756016833221c293a42d884ec55d4025e79273ee7dbde not found: ID does not exist" Oct 31 00:28:42.411247 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.411191 5274 scope.go:122] "RemoveContainer" containerID="b8a139d3b41947671950fac51a9ed64f93590f1f2934855c1b9895e63159d0b5" Oct 31 00:28:42.411372 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.411357 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8a139d3b41947671950fac51a9ed64f93590f1f2934855c1b9895e63159d0b5"} err="failed to get container status \"b8a139d3b41947671950fac51a9ed64f93590f1f2934855c1b9895e63159d0b5\": rpc error: code = NotFound desc = could not find container \"b8a139d3b41947671950fac51a9ed64f93590f1f2934855c1b9895e63159d0b5\": container with ID starting with b8a139d3b41947671950fac51a9ed64f93590f1f2934855c1b9895e63159d0b5 not found: ID does not exist" Oct 31 00:28:42.411372 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.411371 5274 scope.go:122] "RemoveContainer" containerID="756ebf069641c84d785c4e79f4a89a1514e483ab40ff6e741113462d4895edb6" Oct 31 00:28:42.411552 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.411535 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"756ebf069641c84d785c4e79f4a89a1514e483ab40ff6e741113462d4895edb6"} err="failed to get container status \"756ebf069641c84d785c4e79f4a89a1514e483ab40ff6e741113462d4895edb6\": rpc error: code = NotFound desc = could not find container \"756ebf069641c84d785c4e79f4a89a1514e483ab40ff6e741113462d4895edb6\": container with ID starting with 756ebf069641c84d785c4e79f4a89a1514e483ab40ff6e741113462d4895edb6 not found: ID does not exist" Oct 31 00:28:42.411552 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.411551 5274 scope.go:122] "RemoveContainer" containerID="0c0cacbbc110c969a184fbd7d90cb2cab919b1678d367df996c58c1f93665e2f" Oct 31 00:28:42.411716 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.411702 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c0cacbbc110c969a184fbd7d90cb2cab919b1678d367df996c58c1f93665e2f"} err="failed to get container status \"0c0cacbbc110c969a184fbd7d90cb2cab919b1678d367df996c58c1f93665e2f\": rpc error: code = NotFound desc = could not find container \"0c0cacbbc110c969a184fbd7d90cb2cab919b1678d367df996c58c1f93665e2f\": container with ID starting with 0c0cacbbc110c969a184fbd7d90cb2cab919b1678d367df996c58c1f93665e2f not found: ID does not exist" Oct 31 00:28:42.411757 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.411715 5274 scope.go:122] "RemoveContainer" containerID="76493e766a972779c8d756016833221c293a42d884ec55d4025e79273ee7dbde" Oct 31 00:28:42.411891 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.411876 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76493e766a972779c8d756016833221c293a42d884ec55d4025e79273ee7dbde"} err="failed to get container status \"76493e766a972779c8d756016833221c293a42d884ec55d4025e79273ee7dbde\": rpc error: code = NotFound desc = could not find container \"76493e766a972779c8d756016833221c293a42d884ec55d4025e79273ee7dbde\": container with ID starting with 76493e766a972779c8d756016833221c293a42d884ec55d4025e79273ee7dbde not found: ID does not exist" Oct 31 00:28:42.411930 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.411890 5274 scope.go:122] "RemoveContainer" containerID="b675229310452f9ad40962546ac6bf1a78e6463b3594d50d2e90c63e50203df2" Oct 31 00:28:42.415168 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.415147 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:28:42.419321 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.419308 5274 scope.go:122] "RemoveContainer" containerID="b675229310452f9ad40962546ac6bf1a78e6463b3594d50d2e90c63e50203df2" Oct 31 00:28:42.419570 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:28:42.419554 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b675229310452f9ad40962546ac6bf1a78e6463b3594d50d2e90c63e50203df2\": container with ID starting with b675229310452f9ad40962546ac6bf1a78e6463b3594d50d2e90c63e50203df2 not found: ID does not exist" containerID="b675229310452f9ad40962546ac6bf1a78e6463b3594d50d2e90c63e50203df2" Oct 31 00:28:42.419632 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.419575 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b675229310452f9ad40962546ac6bf1a78e6463b3594d50d2e90c63e50203df2"} err="failed to get container status \"b675229310452f9ad40962546ac6bf1a78e6463b3594d50d2e90c63e50203df2\": rpc error: code = NotFound desc = could not find container \"b675229310452f9ad40962546ac6bf1a78e6463b3594d50d2e90c63e50203df2\": container with ID starting with b675229310452f9ad40962546ac6bf1a78e6463b3594d50d2e90c63e50203df2 not found: ID does not exist" Oct 31 00:28:42.426530 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.426511 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-lbhb9"] Oct 31 00:28:42.430924 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.430905 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-lbhb9"] Oct 31 00:28:42.443533 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.443516 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:28:42.448778 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:42.448756 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:28:43.481463 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:43.481433 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17d02fc4-f09b-49a6-8517-36c3abc3ae68" path="/var/lib/kubelet/pods/17d02fc4-f09b-49a6-8517-36c3abc3ae68/volumes" Oct 31 00:28:43.481843 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:43.481809 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6" path="/var/lib/kubelet/pods/72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6/volumes" Oct 31 00:28:43.482331 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:43.482316 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a75a1126-5bff-41dd-a732-5a24e6b4d5e5" path="/var/lib/kubelet/pods/a75a1126-5bff-41dd-a732-5a24e6b4d5e5/volumes" Oct 31 00:28:43.483091 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:43.483077 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b" path="/var/lib/kubelet/pods/b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b/volumes" Oct 31 00:28:43.483433 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:43.483422 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf3c0b8e-98ee-4551-9f95-2b4290d189a2" path="/var/lib/kubelet/pods/bf3c0b8e-98ee-4551-9f95-2b4290d189a2/volumes" Oct 31 00:28:44.538368 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.538335 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-jnvhb"] Oct 31 00:28:44.538732 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.538639 5274 cpu_manager.go:414] "containerMap: removing container" podUID="72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6" containerName="init-config-reloader" Oct 31 00:28:44.538732 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.538650 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6" containerName="init-config-reloader" Oct 31 00:28:44.538732 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.538658 5274 cpu_manager.go:414] "containerMap: removing container" podUID="a75a1126-5bff-41dd-a732-5a24e6b4d5e5" containerName="init-config-reloader" Oct 31 00:28:44.538732 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.538663 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a75a1126-5bff-41dd-a732-5a24e6b4d5e5" containerName="init-config-reloader" Oct 31 00:28:44.538732 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.538670 5274 cpu_manager.go:414] "containerMap: removing container" podUID="72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6" containerName="alertmanager" Oct 31 00:28:44.538732 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.538676 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6" containerName="alertmanager" Oct 31 00:28:44.538732 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.538682 5274 cpu_manager.go:414] "containerMap: removing container" podUID="17d02fc4-f09b-49a6-8517-36c3abc3ae68" containerName="otc-container" Oct 31 00:28:44.538732 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.538686 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17d02fc4-f09b-49a6-8517-36c3abc3ae68" containerName="otc-container" Oct 31 00:28:44.538732 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.538694 5274 cpu_manager.go:414] "containerMap: removing container" podUID="a75a1126-5bff-41dd-a732-5a24e6b4d5e5" containerName="thanos-sidecar" Oct 31 00:28:44.538732 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.538698 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a75a1126-5bff-41dd-a732-5a24e6b4d5e5" containerName="thanos-sidecar" Oct 31 00:28:44.538732 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.538707 5274 cpu_manager.go:414] "containerMap: removing container" podUID="bf3c0b8e-98ee-4551-9f95-2b4290d189a2" containerName="korrel8r" Oct 31 00:28:44.538732 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.538713 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf3c0b8e-98ee-4551-9f95-2b4290d189a2" containerName="korrel8r" Oct 31 00:28:44.538732 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.538721 5274 cpu_manager.go:414] "containerMap: removing container" podUID="b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b" containerName="kube-rbac-proxy" Oct 31 00:28:44.538732 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.538726 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b" containerName="kube-rbac-proxy" Oct 31 00:28:44.538732 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.538731 5274 cpu_manager.go:414] "containerMap: removing container" podUID="72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6" containerName="config-reloader" Oct 31 00:28:44.538732 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.538736 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6" containerName="config-reloader" Oct 31 00:28:44.539210 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.538746 5274 cpu_manager.go:414] "containerMap: removing container" podUID="a75a1126-5bff-41dd-a732-5a24e6b4d5e5" containerName="prometheus" Oct 31 00:28:44.539210 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.538750 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a75a1126-5bff-41dd-a732-5a24e6b4d5e5" containerName="prometheus" Oct 31 00:28:44.539210 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.538760 5274 cpu_manager.go:414] "containerMap: removing container" podUID="a75a1126-5bff-41dd-a732-5a24e6b4d5e5" containerName="config-reloader" Oct 31 00:28:44.539210 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.538764 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a75a1126-5bff-41dd-a732-5a24e6b4d5e5" containerName="config-reloader" Oct 31 00:28:44.539210 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.538811 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="17d02fc4-f09b-49a6-8517-36c3abc3ae68" containerName="otc-container" Oct 31 00:28:44.539210 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.538818 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6" containerName="alertmanager" Oct 31 00:28:44.539210 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.538824 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="a75a1126-5bff-41dd-a732-5a24e6b4d5e5" containerName="thanos-sidecar" Oct 31 00:28:44.539210 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.538829 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf3c0b8e-98ee-4551-9f95-2b4290d189a2" containerName="korrel8r" Oct 31 00:28:44.539210 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.538834 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5b97c7a-e2ac-4e03-8374-92ee0ec0ff0b" containerName="kube-rbac-proxy" Oct 31 00:28:44.539210 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.538839 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="72cfcd9d-cf49-4b29-b378-dec7b4b3dfe6" containerName="config-reloader" Oct 31 00:28:44.539210 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.538845 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="a75a1126-5bff-41dd-a732-5a24e6b4d5e5" containerName="prometheus" Oct 31 00:28:44.539210 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.538851 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="a75a1126-5bff-41dd-a732-5a24e6b4d5e5" containerName="config-reloader" Oct 31 00:28:44.543390 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.543370 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-jnvhb" Oct 31 00:28:44.550888 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.550863 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-jnvhb"] Oct 31 00:28:44.554998 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.554932 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:28:44.554998 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.554976 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:28:44.555165 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.555011 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-9bwnq\"" Oct 31 00:28:44.555165 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.555020 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 31 00:28:44.556144 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.556130 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 31 00:28:44.634158 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.634134 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c6eae400-d4f0-4886-9745-185ffabd1225-tls\") pod \"korrel8r-6644769df5-jnvhb\" (UID: \"c6eae400-d4f0-4886-9745-185ffabd1225\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-jnvhb" Oct 31 00:28:44.634288 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.634160 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c6eae400-d4f0-4886-9745-185ffabd1225-config\") pod \"korrel8r-6644769df5-jnvhb\" (UID: \"c6eae400-d4f0-4886-9745-185ffabd1225\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-jnvhb" Oct 31 00:28:44.634288 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.634196 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snvms\" (UniqueName: \"kubernetes.io/projected/c6eae400-d4f0-4886-9745-185ffabd1225-kube-api-access-snvms\") pod \"korrel8r-6644769df5-jnvhb\" (UID: \"c6eae400-d4f0-4886-9745-185ffabd1225\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-jnvhb" Oct 31 00:28:44.735149 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.735120 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snvms\" (UniqueName: \"kubernetes.io/projected/c6eae400-d4f0-4886-9745-185ffabd1225-kube-api-access-snvms\") pod \"korrel8r-6644769df5-jnvhb\" (UID: \"c6eae400-d4f0-4886-9745-185ffabd1225\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-jnvhb" Oct 31 00:28:44.735280 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.735176 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c6eae400-d4f0-4886-9745-185ffabd1225-tls\") pod \"korrel8r-6644769df5-jnvhb\" (UID: \"c6eae400-d4f0-4886-9745-185ffabd1225\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-jnvhb" Oct 31 00:28:44.735280 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.735193 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c6eae400-d4f0-4886-9745-185ffabd1225-config\") pod \"korrel8r-6644769df5-jnvhb\" (UID: \"c6eae400-d4f0-4886-9745-185ffabd1225\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-jnvhb" Oct 31 00:28:44.735902 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.735875 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c6eae400-d4f0-4886-9745-185ffabd1225-config\") pod \"korrel8r-6644769df5-jnvhb\" (UID: \"c6eae400-d4f0-4886-9745-185ffabd1225\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-jnvhb" Oct 31 00:28:44.738343 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.738320 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c6eae400-d4f0-4886-9745-185ffabd1225-tls\") pod \"korrel8r-6644769df5-jnvhb\" (UID: \"c6eae400-d4f0-4886-9745-185ffabd1225\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-jnvhb" Oct 31 00:28:44.748118 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.748090 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snvms\" (UniqueName: \"kubernetes.io/projected/c6eae400-d4f0-4886-9745-185ffabd1225-kube-api-access-snvms\") pod \"korrel8r-6644769df5-jnvhb\" (UID: \"c6eae400-d4f0-4886-9745-185ffabd1225\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-jnvhb" Oct 31 00:28:44.857265 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:44.857188 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-jnvhb" Oct 31 00:28:45.153353 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.153328 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-jnvhb"] Oct 31 00:28:45.155266 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:28:45.155236 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6eae400_d4f0_4886_9745_185ffabd1225.slice/crio-1c4b721149e5066f3c8c4ba43de1fd394834b47d04c189d096ae4630a610f420 WatchSource:0}: Error finding container 1c4b721149e5066f3c8c4ba43de1fd394834b47d04c189d096ae4630a610f420: Status 404 returned error can't find the container with id 1c4b721149e5066f3c8c4ba43de1fd394834b47d04c189d096ae4630a610f420 Oct 31 00:28:45.360916 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.360879 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-jnvhb" event={"ID":"c6eae400-d4f0-4886-9745-185ffabd1225","Type":"ContainerStarted","Data":"2044943061abdb1be52c557fc8bc7a80484e063853de9e78e1f5f0adccad5932"} Oct 31 00:28:45.361108 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.360941 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-jnvhb" event={"ID":"c6eae400-d4f0-4886-9745-185ffabd1225","Type":"ContainerStarted","Data":"1c4b721149e5066f3c8c4ba43de1fd394834b47d04c189d096ae4630a610f420"} Oct 31 00:28:45.361108 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.361016 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-jnvhb" Oct 31 00:28:45.383738 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.383680 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-6644769df5-jnvhb" podStartSLOduration=1.383665653 podStartE2EDuration="1.383665653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:45.382905313 +0000 UTC m=+1762.019724030" watchObservedRunningTime="2026-10-31 00:28:45.383665653 +0000 UTC m=+1762.020484373" Oct 31 00:28:45.783920 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.783890 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:28:45.787700 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.787678 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:28:45.791868 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.791826 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 31 00:28:45.793240 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.793224 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-hsfnc\"" Oct 31 00:28:45.793692 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.793677 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\"" Oct 31 00:28:45.797157 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.797137 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\"" Oct 31 00:28:45.803464 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.803442 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:28:45.817333 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.817310 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\"" Oct 31 00:28:45.845338 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.845314 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:28:45.845443 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.845343 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:28:45.845443 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.845369 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:28:45.845443 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.845389 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xjz5\" (UniqueName: \"kubernetes.io/projected/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-kube-api-access-9xjz5\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:28:45.845668 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.845505 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:28:45.845668 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.845582 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:28:45.845668 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.845634 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:28:45.845827 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.845750 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:28:45.894125 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.894101 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\"" Oct 31 00:28:45.894252 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.894236 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\"" Oct 31 00:28:45.894348 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.894271 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 31 00:28:45.947077 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.947054 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:28:45.947202 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.947088 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:28:45.947202 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.947105 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xjz5\" (UniqueName: \"kubernetes.io/projected/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-kube-api-access-9xjz5\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:28:45.947202 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.947123 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:28:45.947202 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.947140 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:28:45.947202 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.947160 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:28:45.947468 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.947224 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:28:45.947468 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.947279 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:28:45.948003 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.947978 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:28:45.948120 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.948016 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:28:45.948120 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.948040 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:28:45.948120 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.948019 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:28:45.949782 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.949758 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:28:45.949869 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.949839 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:28:45.949869 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.949847 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:28:45.959785 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:45.959765 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xjz5\" (UniqueName: \"kubernetes.io/projected/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-kube-api-access-9xjz5\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:28:49.873891 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:49.873859 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:28:49.881077 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:49.881049 5274 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:28:49.974538 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:49.974509 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:28:49.974673 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:49.974607 5274 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:28:49.978340 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:49.978321 5274 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") DevicePath \"csi-9ce3c80b5d5deac9dfdb953db4fb017080b654a85db76a2937e67cd541d89057\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:28:50.058974 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:50.058937 5274 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/151619720cb39d40781bf258206fb0be746b77e5f55879e189e4cf0d6e46dfb0/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:28:50.074593 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:50.074571 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:28:50.307703 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:50.307630 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:28:50.589175 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:50.589145 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:28:50.591157 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:28:50.591133 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb13f2f_e1f0_471c_9d7f_eb74c7a979ce.slice/crio-c803e1cf03fc71f61254d09e91bebe72c54f4f80fff082b8ff40cb6c57b5460f WatchSource:0}: Error finding container c803e1cf03fc71f61254d09e91bebe72c54f4f80fff082b8ff40cb6c57b5460f: Status 404 returned error can't find the container with id c803e1cf03fc71f61254d09e91bebe72c54f4f80fff082b8ff40cb6c57b5460f Oct 31 00:28:50.898344 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:28:50.898313 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:28:51.396709 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:51.396678 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce","Type":"ContainerStarted","Data":"0250b5b79b92d6e85d4d2317fc2f24505755e9d19b7f04b8ef166c6e2f69016f"} Oct 31 00:28:51.396900 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:51.396729 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce","Type":"ContainerStarted","Data":"ce8c7279afe68b360f8966e9dc878c694617b8b10f031af61924c905789a56bc"} Oct 31 00:28:51.396900 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:51.396755 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce","Type":"ContainerStarted","Data":"e3be14006ee8c985582fd577d17fe0660909251b8e33b504efefbb2e96b93120"} Oct 31 00:28:51.396900 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:51.396774 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce","Type":"ContainerStarted","Data":"c803e1cf03fc71f61254d09e91bebe72c54f4f80fff082b8ff40cb6c57b5460f"} Oct 31 00:28:51.396900 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:51.396852 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:28:51.396900 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:51.396902 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:28:51.397176 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:51.396926 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:28:51.399592 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:51.399572 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:28:51.405553 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:51.405534 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:28:51.425498 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:51.425457 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=6.42544645 podStartE2EDuration="6.42544645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:51.423869786 +0000 UTC m=+1768.060688501" watchObservedRunningTime="2026-10-31 00:28:51.42544645 +0000 UTC m=+1768.062265166" Oct 31 00:28:56.368745 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:28:56.368711 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-jnvhb" Oct 31 00:29:13.408600 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:13.408553 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:29:20.898827 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:29:20.898787 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:29:23.408372 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:23.408324 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:29:33.408568 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:33.408532 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:34.982305 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:34.982269 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-jnvhb"] Oct 31 00:29:34.982727 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:34.982492 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-6644769df5-jnvhb" podUID="c6eae400-d4f0-4886-9745-185ffabd1225" containerName="korrel8r" containerID="cri-o://2044943061abdb1be52c557fc8bc7a80484e063853de9e78e1f5f0adccad5932" gracePeriod=30 Oct 31 00:29:35.217712 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:35.217672 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:29:35.218029 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:35.217988 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce" containerName="tempo" containerID="cri-o://e3be14006ee8c985582fd577d17fe0660909251b8e33b504efefbb2e96b93120" gracePeriod=30 Oct 31 00:29:35.218161 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:35.218034 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce" containerName="tempo-gateway-opa" containerID="cri-o://0250b5b79b92d6e85d4d2317fc2f24505755e9d19b7f04b8ef166c6e2f69016f" gracePeriod=30 Oct 31 00:29:35.218231 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:35.218151 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce" containerName="tempo-gateway" containerID="cri-o://ce8c7279afe68b360f8966e9dc878c694617b8b10f031af61924c905789a56bc" gracePeriod=30 Oct 31 00:29:35.349880 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:35.349856 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-jnvhb" Oct 31 00:29:35.454596 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:35.454573 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6eae400-d4f0-4886-9745-185ffabd1225-tls\" (UniqueName: \"kubernetes.io/secret/c6eae400-d4f0-4886-9745-185ffabd1225-tls\") pod \"c6eae400-d4f0-4886-9745-185ffabd1225\" (UID: \"c6eae400-d4f0-4886-9745-185ffabd1225\") " Oct 31 00:29:35.454691 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:35.454624 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6eae400-d4f0-4886-9745-185ffabd1225-config\" (UniqueName: \"kubernetes.io/configmap/c6eae400-d4f0-4886-9745-185ffabd1225-config\") pod \"c6eae400-d4f0-4886-9745-185ffabd1225\" (UID: \"c6eae400-d4f0-4886-9745-185ffabd1225\") " Oct 31 00:29:35.454737 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:35.454716 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6eae400-d4f0-4886-9745-185ffabd1225-kube-api-access-snvms\" (UniqueName: \"kubernetes.io/projected/c6eae400-d4f0-4886-9745-185ffabd1225-kube-api-access-snvms\") pod \"c6eae400-d4f0-4886-9745-185ffabd1225\" (UID: \"c6eae400-d4f0-4886-9745-185ffabd1225\") " Oct 31 00:29:35.455048 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:35.455025 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6eae400-d4f0-4886-9745-185ffabd1225-config" pod "c6eae400-d4f0-4886-9745-185ffabd1225" (UID: "c6eae400-d4f0-4886-9745-185ffabd1225"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:35.456553 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:35.456534 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6eae400-d4f0-4886-9745-185ffabd1225-tls" pod "c6eae400-d4f0-4886-9745-185ffabd1225" (UID: "c6eae400-d4f0-4886-9745-185ffabd1225"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:35.456695 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:35.456673 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6eae400-d4f0-4886-9745-185ffabd1225-kube-api-access-snvms" pod "c6eae400-d4f0-4886-9745-185ffabd1225" (UID: "c6eae400-d4f0-4886-9745-185ffabd1225"). InnerVolumeSpecName "kube-api-access-snvms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:35.555943 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:35.555909 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snvms\" (UniqueName: \"kubernetes.io/projected/c6eae400-d4f0-4886-9745-185ffabd1225-kube-api-access-snvms\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:35.555943 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:35.555937 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c6eae400-d4f0-4886-9745-185ffabd1225-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:35.555943 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:35.555950 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c6eae400-d4f0-4886-9745-185ffabd1225-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:35.615928 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:35.615887 5274 generic.go:356] "Generic (PLEG): container finished" podID="3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce" containerID="0250b5b79b92d6e85d4d2317fc2f24505755e9d19b7f04b8ef166c6e2f69016f" exitCode=0 Oct 31 00:29:35.615928 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:35.615913 5274 generic.go:356] "Generic (PLEG): container finished" podID="3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce" containerID="e3be14006ee8c985582fd577d17fe0660909251b8e33b504efefbb2e96b93120" exitCode=0 Oct 31 00:29:35.616147 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:35.615973 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce","Type":"ContainerDied","Data":"0250b5b79b92d6e85d4d2317fc2f24505755e9d19b7f04b8ef166c6e2f69016f"} Oct 31 00:29:35.616147 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:35.616022 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce","Type":"ContainerDied","Data":"e3be14006ee8c985582fd577d17fe0660909251b8e33b504efefbb2e96b93120"} Oct 31 00:29:35.617103 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:35.617084 5274 generic.go:356] "Generic (PLEG): container finished" podID="c6eae400-d4f0-4886-9745-185ffabd1225" containerID="2044943061abdb1be52c557fc8bc7a80484e063853de9e78e1f5f0adccad5932" exitCode=0 Oct 31 00:29:35.617171 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:35.617130 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-jnvhb" Oct 31 00:29:35.617224 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:35.617131 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-jnvhb" event={"ID":"c6eae400-d4f0-4886-9745-185ffabd1225","Type":"ContainerDied","Data":"2044943061abdb1be52c557fc8bc7a80484e063853de9e78e1f5f0adccad5932"} Oct 31 00:29:35.617264 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:35.617245 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-jnvhb" event={"ID":"c6eae400-d4f0-4886-9745-185ffabd1225","Type":"ContainerDied","Data":"1c4b721149e5066f3c8c4ba43de1fd394834b47d04c189d096ae4630a610f420"} Oct 31 00:29:35.617308 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:35.617268 5274 scope.go:122] "RemoveContainer" containerID="2044943061abdb1be52c557fc8bc7a80484e063853de9e78e1f5f0adccad5932" Oct 31 00:29:35.625633 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:35.625614 5274 scope.go:122] "RemoveContainer" containerID="2044943061abdb1be52c557fc8bc7a80484e063853de9e78e1f5f0adccad5932" Oct 31 00:29:35.625884 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:29:35.625864 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2044943061abdb1be52c557fc8bc7a80484e063853de9e78e1f5f0adccad5932\": container with ID starting with 2044943061abdb1be52c557fc8bc7a80484e063853de9e78e1f5f0adccad5932 not found: ID does not exist" containerID="2044943061abdb1be52c557fc8bc7a80484e063853de9e78e1f5f0adccad5932" Oct 31 00:29:35.625998 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:35.625894 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2044943061abdb1be52c557fc8bc7a80484e063853de9e78e1f5f0adccad5932"} err="failed to get container status \"2044943061abdb1be52c557fc8bc7a80484e063853de9e78e1f5f0adccad5932\": rpc error: code = NotFound desc = could not find container \"2044943061abdb1be52c557fc8bc7a80484e063853de9e78e1f5f0adccad5932\": container with ID starting with 2044943061abdb1be52c557fc8bc7a80484e063853de9e78e1f5f0adccad5932 not found: ID does not exist" Oct 31 00:29:35.636380 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:35.636355 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-jnvhb"] Oct 31 00:29:35.641057 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:35.641028 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-jnvhb"] Oct 31 00:29:36.476356 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.476334 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:36.623764 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.623732 5274 generic.go:356] "Generic (PLEG): container finished" podID="3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce" containerID="ce8c7279afe68b360f8966e9dc878c694617b8b10f031af61924c905789a56bc" exitCode=0 Oct 31 00:29:36.623914 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.623793 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:36.623914 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.623808 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce","Type":"ContainerDied","Data":"ce8c7279afe68b360f8966e9dc878c694617b8b10f031af61924c905789a56bc"} Oct 31 00:29:36.623914 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.623872 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce","Type":"ContainerDied","Data":"c803e1cf03fc71f61254d09e91bebe72c54f4f80fff082b8ff40cb6c57b5460f"} Oct 31 00:29:36.623914 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.623907 5274 scope.go:122] "RemoveContainer" containerID="0250b5b79b92d6e85d4d2317fc2f24505755e9d19b7f04b8ef166c6e2f69016f" Oct 31 00:29:36.632029 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.632012 5274 scope.go:122] "RemoveContainer" containerID="ce8c7279afe68b360f8966e9dc878c694617b8b10f031af61924c905789a56bc" Oct 31 00:29:36.639415 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.639398 5274 scope.go:122] "RemoveContainer" containerID="e3be14006ee8c985582fd577d17fe0660909251b8e33b504efefbb2e96b93120" Oct 31 00:29:36.646529 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.646512 5274 scope.go:122] "RemoveContainer" containerID="0250b5b79b92d6e85d4d2317fc2f24505755e9d19b7f04b8ef166c6e2f69016f" Oct 31 00:29:36.646815 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:29:36.646794 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0250b5b79b92d6e85d4d2317fc2f24505755e9d19b7f04b8ef166c6e2f69016f\": container with ID starting with 0250b5b79b92d6e85d4d2317fc2f24505755e9d19b7f04b8ef166c6e2f69016f not found: ID does not exist" containerID="0250b5b79b92d6e85d4d2317fc2f24505755e9d19b7f04b8ef166c6e2f69016f" Oct 31 00:29:36.646899 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.646823 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0250b5b79b92d6e85d4d2317fc2f24505755e9d19b7f04b8ef166c6e2f69016f"} err="failed to get container status \"0250b5b79b92d6e85d4d2317fc2f24505755e9d19b7f04b8ef166c6e2f69016f\": rpc error: code = NotFound desc = could not find container \"0250b5b79b92d6e85d4d2317fc2f24505755e9d19b7f04b8ef166c6e2f69016f\": container with ID starting with 0250b5b79b92d6e85d4d2317fc2f24505755e9d19b7f04b8ef166c6e2f69016f not found: ID does not exist" Oct 31 00:29:36.646899 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.646837 5274 scope.go:122] "RemoveContainer" containerID="ce8c7279afe68b360f8966e9dc878c694617b8b10f031af61924c905789a56bc" Oct 31 00:29:36.647092 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:29:36.647073 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce8c7279afe68b360f8966e9dc878c694617b8b10f031af61924c905789a56bc\": container with ID starting with ce8c7279afe68b360f8966e9dc878c694617b8b10f031af61924c905789a56bc not found: ID does not exist" containerID="ce8c7279afe68b360f8966e9dc878c694617b8b10f031af61924c905789a56bc" Oct 31 00:29:36.647134 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.647097 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce8c7279afe68b360f8966e9dc878c694617b8b10f031af61924c905789a56bc"} err="failed to get container status \"ce8c7279afe68b360f8966e9dc878c694617b8b10f031af61924c905789a56bc\": rpc error: code = NotFound desc = could not find container \"ce8c7279afe68b360f8966e9dc878c694617b8b10f031af61924c905789a56bc\": container with ID starting with ce8c7279afe68b360f8966e9dc878c694617b8b10f031af61924c905789a56bc not found: ID does not exist" Oct 31 00:29:36.647134 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.647111 5274 scope.go:122] "RemoveContainer" containerID="e3be14006ee8c985582fd577d17fe0660909251b8e33b504efefbb2e96b93120" Oct 31 00:29:36.647359 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:29:36.647339 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3be14006ee8c985582fd577d17fe0660909251b8e33b504efefbb2e96b93120\": container with ID starting with e3be14006ee8c985582fd577d17fe0660909251b8e33b504efefbb2e96b93120 not found: ID does not exist" containerID="e3be14006ee8c985582fd577d17fe0660909251b8e33b504efefbb2e96b93120" Oct 31 00:29:36.647419 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.647377 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3be14006ee8c985582fd577d17fe0660909251b8e33b504efefbb2e96b93120"} err="failed to get container status \"e3be14006ee8c985582fd577d17fe0660909251b8e33b504efefbb2e96b93120\": rpc error: code = NotFound desc = could not find container \"e3be14006ee8c985582fd577d17fe0660909251b8e33b504efefbb2e96b93120\": container with ID starting with e3be14006ee8c985582fd577d17fe0660909251b8e33b504efefbb2e96b93120 not found: ID does not exist" Oct 31 00:29:36.667068 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.667045 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-kube-api-access-9xjz5\" (UniqueName: \"kubernetes.io/projected/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-kube-api-access-9xjz5\") pod \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") " Oct 31 00:29:36.667167 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.667093 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") " Oct 31 00:29:36.667167 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.667132 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-tempo-data-science-tempomonolithic-mtls\") pod \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") " Oct 31 00:29:36.667277 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.667164 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-gateway-tenants\") pod \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") " Oct 31 00:29:36.667277 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.667190 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-gateway-rbac\") pod \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") " Oct 31 00:29:36.667414 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.667302 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") " Oct 31 00:29:36.667414 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.667337 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") " Oct 31 00:29:36.667414 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.667392 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-tempo-conf\") pod \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") " Oct 31 00:29:36.667560 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.667447 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-tempo-data-science-tempomonolithic-ca-bundle\") pod \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\" (UID: \"3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce\") " Oct 31 00:29:36.667692 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.667608 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-gateway-rbac" pod "3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce" (UID: "3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:36.667759 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.667689 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-tempo-data-science-tempomonolithic-serving-cabundle" pod "3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce" (UID: "3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:36.667815 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.667753 5274 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-gateway-rbac\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:36.668094 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.668072 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-tempo-data-science-tempomonolithic-ca-bundle" pod "3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce" (UID: "3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:36.668211 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.668081 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-tempo-conf" pod "3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce" (UID: "3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:36.669316 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.669285 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-tempo-data-science-tempomonolithic-mtls" pod "3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce" (UID: "3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:36.669576 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.669553 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-kube-api-access-9xjz5" pod "3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce" (UID: "3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce"). InnerVolumeSpecName "kube-api-access-9xjz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:36.669709 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.669662 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce" (UID: "3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:36.669709 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.669675 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-gateway-tenants" pod "3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce" (UID: "3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:36.670151 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.670133 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062" pod "3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce" (UID: "3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce"). InnerVolumeSpecName "pvc-4491ac70-4597-43a7-a792-f0813314e433". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:29:36.768496 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.768461 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:36.768496 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.768481 5274 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-gateway-tenants\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:36.768617 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.768505 5274 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" " Oct 31 00:29:36.768617 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.768515 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:36.768617 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.768526 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-tempo-conf\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:36.768617 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.768535 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:36.768617 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.768543 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xjz5\" (UniqueName: \"kubernetes.io/projected/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-kube-api-access-9xjz5\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:36.768617 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.768553 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:36.776583 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.776566 5274 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-4491ac70-4597-43a7-a792-f0813314e433" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062") on node "ip-10-0-129-70.us-east-2.compute.internal" Oct 31 00:29:36.868880 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.868856 5274 reconciler_common.go:299] "Volume detached for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"csi-9ce3c80b5d5deac9dfdb953db4fb017080b654a85db76a2937e67cd541d89057\"" Oct 31 00:29:36.947694 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.947672 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:29:36.955326 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:36.955274 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:29:37.398265 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:37.398228 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.134.0.67:8083/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:29:37.398466 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:37.398245 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce" containerName="tempo-gateway" probeResult="failure" output="Get \"https://10.134.0.67:8081/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:29:37.481165 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:37.481136 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce" path="/var/lib/kubelet/pods/3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce/volumes" Oct 31 00:29:37.481586 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:37.481573 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6eae400-d4f0-4886-9745-185ffabd1225" path="/var/lib/kubelet/pods/c6eae400-d4f0-4886-9745-185ffabd1225/volumes" Oct 31 00:29:37.946163 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:37.946135 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-45gq9"] Oct 31 00:29:37.946473 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:37.946460 5274 cpu_manager.go:414] "containerMap: removing container" podUID="3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce" containerName="tempo" Oct 31 00:29:37.946517 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:37.946474 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce" containerName="tempo" Oct 31 00:29:37.946517 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:37.946489 5274 cpu_manager.go:414] "containerMap: removing container" podUID="3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce" containerName="tempo-gateway" Oct 31 00:29:37.946517 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:37.946494 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce" containerName="tempo-gateway" Oct 31 00:29:37.946517 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:37.946503 5274 cpu_manager.go:414] "containerMap: removing container" podUID="3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce" containerName="tempo-gateway-opa" Oct 31 00:29:37.946517 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:37.946509 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce" containerName="tempo-gateway-opa" Oct 31 00:29:37.946659 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:37.946531 5274 cpu_manager.go:414] "containerMap: removing container" podUID="c6eae400-d4f0-4886-9745-185ffabd1225" containerName="korrel8r" Oct 31 00:29:37.946659 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:37.946536 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6eae400-d4f0-4886-9745-185ffabd1225" containerName="korrel8r" Oct 31 00:29:37.946659 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:37.946583 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6eae400-d4f0-4886-9745-185ffabd1225" containerName="korrel8r" Oct 31 00:29:37.946659 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:37.946591 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce" containerName="tempo" Oct 31 00:29:37.946659 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:37.946598 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce" containerName="tempo-gateway" Oct 31 00:29:37.946659 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:37.946606 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bb13f2f-e1f0-471c-9d7f-eb74c7a979ce" containerName="tempo-gateway-opa" Oct 31 00:29:37.951437 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:37.951420 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-45gq9" Oct 31 00:29:37.958072 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:37.957934 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 31 00:29:37.958072 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:37.958069 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-nswrj\"" Oct 31 00:29:37.958204 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:37.958118 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:29:37.958674 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:37.958643 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 31 00:29:37.958787 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:37.958724 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:29:37.959345 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:37.959323 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-45gq9"] Oct 31 00:29:37.977425 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:37.977399 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7lt6\" (UniqueName: \"kubernetes.io/projected/5baf1c47-1d9f-4d68-93c8-7061a486469f-kube-api-access-d7lt6\") pod \"korrel8r-6644769df5-45gq9\" (UID: \"5baf1c47-1d9f-4d68-93c8-7061a486469f\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-45gq9" Oct 31 00:29:37.977557 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:37.977444 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5baf1c47-1d9f-4d68-93c8-7061a486469f-tls\") pod \"korrel8r-6644769df5-45gq9\" (UID: \"5baf1c47-1d9f-4d68-93c8-7061a486469f\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-45gq9" Oct 31 00:29:37.977622 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:37.977576 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5baf1c47-1d9f-4d68-93c8-7061a486469f-config\") pod \"korrel8r-6644769df5-45gq9\" (UID: \"5baf1c47-1d9f-4d68-93c8-7061a486469f\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-45gq9" Oct 31 00:29:38.078165 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:38.078129 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5baf1c47-1d9f-4d68-93c8-7061a486469f-tls\") pod \"korrel8r-6644769df5-45gq9\" (UID: \"5baf1c47-1d9f-4d68-93c8-7061a486469f\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-45gq9" Oct 31 00:29:38.078328 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:38.078222 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5baf1c47-1d9f-4d68-93c8-7061a486469f-config\") pod \"korrel8r-6644769df5-45gq9\" (UID: \"5baf1c47-1d9f-4d68-93c8-7061a486469f\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-45gq9" Oct 31 00:29:38.078328 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:38.078263 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7lt6\" (UniqueName: \"kubernetes.io/projected/5baf1c47-1d9f-4d68-93c8-7061a486469f-kube-api-access-d7lt6\") pod \"korrel8r-6644769df5-45gq9\" (UID: \"5baf1c47-1d9f-4d68-93c8-7061a486469f\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-45gq9" Oct 31 00:29:38.078328 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:29:38.078272 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/korrel8r-tls: secret "korrel8r-tls" not found Oct 31 00:29:38.078328 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:29:38.078328 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5baf1c47-1d9f-4d68-93c8-7061a486469f-tls podName:5baf1c47-1d9f-4d68-93c8-7061a486469f nodeName:}" failed. No retries permitted until 2026-10-31 00:29:38.578313427 +0000 UTC m=+1815.215132121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls" (UniqueName: "kubernetes.io/secret/5baf1c47-1d9f-4d68-93c8-7061a486469f-tls") pod "korrel8r-6644769df5-45gq9" (UID: "5baf1c47-1d9f-4d68-93c8-7061a486469f") : secret "korrel8r-tls" not found Oct 31 00:29:38.078928 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:38.078908 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5baf1c47-1d9f-4d68-93c8-7061a486469f-config\") pod \"korrel8r-6644769df5-45gq9\" (UID: \"5baf1c47-1d9f-4d68-93c8-7061a486469f\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-45gq9" Oct 31 00:29:38.090675 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:38.090647 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7lt6\" (UniqueName: \"kubernetes.io/projected/5baf1c47-1d9f-4d68-93c8-7061a486469f-kube-api-access-d7lt6\") pod \"korrel8r-6644769df5-45gq9\" (UID: \"5baf1c47-1d9f-4d68-93c8-7061a486469f\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-45gq9" Oct 31 00:29:38.582334 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:38.582300 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5baf1c47-1d9f-4d68-93c8-7061a486469f-tls\") pod \"korrel8r-6644769df5-45gq9\" (UID: \"5baf1c47-1d9f-4d68-93c8-7061a486469f\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-45gq9" Oct 31 00:29:38.584621 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:38.584598 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5baf1c47-1d9f-4d68-93c8-7061a486469f-tls\") pod \"korrel8r-6644769df5-45gq9\" (UID: \"5baf1c47-1d9f-4d68-93c8-7061a486469f\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-45gq9" Oct 31 00:29:38.864538 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:38.864457 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-45gq9" Oct 31 00:29:39.140314 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.140291 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-45gq9"] Oct 31 00:29:39.142450 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:29:39.142422 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5baf1c47_1d9f_4d68_93c8_7061a486469f.slice/crio-84f17ae1db3fa7d3f4ee8b16d07f4460a481cc85f51722374802b3c46fa00c97 WatchSource:0}: Error finding container 84f17ae1db3fa7d3f4ee8b16d07f4460a481cc85f51722374802b3c46fa00c97: Status 404 returned error can't find the container with id 84f17ae1db3fa7d3f4ee8b16d07f4460a481cc85f51722374802b3c46fa00c97 Oct 31 00:29:39.419757 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.419686 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:29:39.423481 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.423460 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:39.428561 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.428518 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\"" Oct 31 00:29:39.428786 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.428772 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 31 00:29:39.429036 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.429013 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-mcp52\"" Oct 31 00:29:39.429206 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.429191 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\"" Oct 31 00:29:39.429259 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.429241 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\"" Oct 31 00:29:39.429419 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.429406 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 31 00:29:39.429459 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.429416 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\"" Oct 31 00:29:39.429572 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.429558 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\"" Oct 31 00:29:39.438292 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.438270 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:29:39.591341 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.591313 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/84158686-4b22-4c87-ad4b-19a969a156fd-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:39.591718 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.591350 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/84158686-4b22-4c87-ad4b-19a969a156fd-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:39.591718 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.591367 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q75wt\" (UniqueName: \"kubernetes.io/projected/84158686-4b22-4c87-ad4b-19a969a156fd-kube-api-access-q75wt\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:39.591718 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.591385 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/84158686-4b22-4c87-ad4b-19a969a156fd-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:39.591718 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.591481 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/84158686-4b22-4c87-ad4b-19a969a156fd-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:39.591718 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.591532 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84158686-4b22-4c87-ad4b-19a969a156fd-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:39.591718 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.591636 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/84158686-4b22-4c87-ad4b-19a969a156fd-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:39.591718 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.591668 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/84158686-4b22-4c87-ad4b-19a969a156fd-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:39.643104 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.643078 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-45gq9" event={"ID":"5baf1c47-1d9f-4d68-93c8-7061a486469f","Type":"ContainerStarted","Data":"4d1372b886b8cee85033d37931df93f23f9fd0f323a7549aa134701c990479c3"} Oct 31 00:29:39.643239 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.643123 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-45gq9" event={"ID":"5baf1c47-1d9f-4d68-93c8-7061a486469f","Type":"ContainerStarted","Data":"84f17ae1db3fa7d3f4ee8b16d07f4460a481cc85f51722374802b3c46fa00c97"} Oct 31 00:29:39.643239 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.643185 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-45gq9" Oct 31 00:29:39.661825 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.661780 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-6644769df5-45gq9" podStartSLOduration=2.661764454 podStartE2EDuration="2.661764454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:39.661245036 +0000 UTC m=+1816.298063760" watchObservedRunningTime="2026-10-31 00:29:39.661764454 +0000 UTC m=+1816.298583186" Oct 31 00:29:39.692353 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.692285 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/84158686-4b22-4c87-ad4b-19a969a156fd-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:39.692353 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.692328 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/84158686-4b22-4c87-ad4b-19a969a156fd-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:39.692530 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.692352 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q75wt\" (UniqueName: \"kubernetes.io/projected/84158686-4b22-4c87-ad4b-19a969a156fd-kube-api-access-q75wt\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:39.692530 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.692380 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/84158686-4b22-4c87-ad4b-19a969a156fd-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:39.692530 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.692414 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/84158686-4b22-4c87-ad4b-19a969a156fd-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:39.692530 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.692480 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84158686-4b22-4c87-ad4b-19a969a156fd-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:39.692733 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.692528 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/84158686-4b22-4c87-ad4b-19a969a156fd-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:39.692733 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.692565 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/84158686-4b22-4c87-ad4b-19a969a156fd-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:39.693184 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.693157 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/84158686-4b22-4c87-ad4b-19a969a156fd-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:39.693330 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.693164 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/84158686-4b22-4c87-ad4b-19a969a156fd-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:39.693409 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.693333 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/84158686-4b22-4c87-ad4b-19a969a156fd-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:39.693409 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.693368 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84158686-4b22-4c87-ad4b-19a969a156fd-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:39.695182 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.695159 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/84158686-4b22-4c87-ad4b-19a969a156fd-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:39.695463 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.695442 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/84158686-4b22-4c87-ad4b-19a969a156fd-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:39.695526 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.695503 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/84158686-4b22-4c87-ad4b-19a969a156fd-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:39.708247 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:39.708228 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q75wt\" (UniqueName: \"kubernetes.io/projected/84158686-4b22-4c87-ad4b-19a969a156fd-kube-api-access-q75wt\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:40.702126 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:40.702091 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:40.710874 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:40.710849 5274 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:40.802631 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:40.802603 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:40.802769 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:40.802712 5274 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:40.806354 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:40.806337 5274 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") DevicePath \"csi-9ce3c80b5d5deac9dfdb953db4fb017080b654a85db76a2937e67cd541d89057\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:40.894343 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:40.894302 5274 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/151619720cb39d40781bf258206fb0be746b77e5f55879e189e4cf0d6e46dfb0/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:40.909950 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:40.909921 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:40.939583 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:40.939559 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:41.227285 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:41.227257 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:29:41.228574 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:29:41.228549 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84158686_4b22_4c87_ad4b_19a969a156fd.slice/crio-33ad5ae8d283f8d7dd970978f162a6f63d667d3a808bce7d309ae4efeb7bbee8 WatchSource:0}: Error finding container 33ad5ae8d283f8d7dd970978f162a6f63d667d3a808bce7d309ae4efeb7bbee8: Status 404 returned error can't find the container with id 33ad5ae8d283f8d7dd970978f162a6f63d667d3a808bce7d309ae4efeb7bbee8 Oct 31 00:29:41.657450 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:41.657406 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"84158686-4b22-4c87-ad4b-19a969a156fd","Type":"ContainerStarted","Data":"1dc5f97d29697d0c0a37a7fa4fda29b5f9e51fd614176bb3d81dbf84ac755a43"} Oct 31 00:29:41.657609 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:41.657463 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"84158686-4b22-4c87-ad4b-19a969a156fd","Type":"ContainerStarted","Data":"12746428d807a4221d75dc24ceae9d336797cee9ed6a9e56f8780aa736fdac27"} Oct 31 00:29:41.657609 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:41.657481 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"84158686-4b22-4c87-ad4b-19a969a156fd","Type":"ContainerStarted","Data":"f8f895b1e4d7c4cfba87d44a55e8443931cb4db38bd3558676817955dadf0ba6"} Oct 31 00:29:41.657609 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:41.657495 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"84158686-4b22-4c87-ad4b-19a969a156fd","Type":"ContainerStarted","Data":"33ad5ae8d283f8d7dd970978f162a6f63d667d3a808bce7d309ae4efeb7bbee8"} Oct 31 00:29:41.657609 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:41.657513 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:41.659855 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:41.659836 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:41.687363 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:41.687321 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=2.687306743 podStartE2EDuration="2.687306743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:41.68460395 +0000 UTC m=+1818.321422666" watchObservedRunningTime="2026-10-31 00:29:41.687306743 +0000 UTC m=+1818.324125460" Oct 31 00:29:42.662991 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:42.662939 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:42.663453 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:42.663011 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:42.670814 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:42.670792 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:50.653659 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:50.653624 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-45gq9" Oct 31 00:29:50.898748 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:29:50.898713 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:29:51.886455 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:51.886421 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-45gq9"] Oct 31 00:29:51.886867 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:51.886584 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-6644769df5-45gq9" podUID="5baf1c47-1d9f-4d68-93c8-7061a486469f" containerName="korrel8r" containerID="cri-o://4d1372b886b8cee85033d37931df93f23f9fd0f323a7549aa134701c990479c3" gracePeriod=30 Oct 31 00:29:52.149233 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:52.149204 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:29:52.149706 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:52.149454 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="84158686-4b22-4c87-ad4b-19a969a156fd" containerName="tempo" containerID="cri-o://f8f895b1e4d7c4cfba87d44a55e8443931cb4db38bd3558676817955dadf0ba6" gracePeriod=30 Oct 31 00:29:52.149706 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:52.149525 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="84158686-4b22-4c87-ad4b-19a969a156fd" containerName="tempo-gateway-opa" containerID="cri-o://1dc5f97d29697d0c0a37a7fa4fda29b5f9e51fd614176bb3d81dbf84ac755a43" gracePeriod=30 Oct 31 00:29:52.149706 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:52.149607 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="84158686-4b22-4c87-ad4b-19a969a156fd" containerName="tempo-gateway" containerID="cri-o://12746428d807a4221d75dc24ceae9d336797cee9ed6a9e56f8780aa736fdac27" gracePeriod=30 Oct 31 00:29:52.178546 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:52.178519 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-45gq9" Oct 31 00:29:52.298902 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:52.298869 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5baf1c47-1d9f-4d68-93c8-7061a486469f-kube-api-access-d7lt6\" (UniqueName: \"kubernetes.io/projected/5baf1c47-1d9f-4d68-93c8-7061a486469f-kube-api-access-d7lt6\") pod \"5baf1c47-1d9f-4d68-93c8-7061a486469f\" (UID: \"5baf1c47-1d9f-4d68-93c8-7061a486469f\") " Oct 31 00:29:52.299031 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:52.298911 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5baf1c47-1d9f-4d68-93c8-7061a486469f-tls\" (UniqueName: \"kubernetes.io/secret/5baf1c47-1d9f-4d68-93c8-7061a486469f-tls\") pod \"5baf1c47-1d9f-4d68-93c8-7061a486469f\" (UID: \"5baf1c47-1d9f-4d68-93c8-7061a486469f\") " Oct 31 00:29:52.299073 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:52.299029 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5baf1c47-1d9f-4d68-93c8-7061a486469f-config\" (UniqueName: \"kubernetes.io/configmap/5baf1c47-1d9f-4d68-93c8-7061a486469f-config\") pod \"5baf1c47-1d9f-4d68-93c8-7061a486469f\" (UID: \"5baf1c47-1d9f-4d68-93c8-7061a486469f\") " Oct 31 00:29:52.299364 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:52.299337 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5baf1c47-1d9f-4d68-93c8-7061a486469f-config" pod "5baf1c47-1d9f-4d68-93c8-7061a486469f" (UID: "5baf1c47-1d9f-4d68-93c8-7061a486469f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:52.300658 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:52.300641 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5baf1c47-1d9f-4d68-93c8-7061a486469f-tls" pod "5baf1c47-1d9f-4d68-93c8-7061a486469f" (UID: "5baf1c47-1d9f-4d68-93c8-7061a486469f"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:52.300740 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:52.300727 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5baf1c47-1d9f-4d68-93c8-7061a486469f-kube-api-access-d7lt6" pod "5baf1c47-1d9f-4d68-93c8-7061a486469f" (UID: "5baf1c47-1d9f-4d68-93c8-7061a486469f"). InnerVolumeSpecName "kube-api-access-d7lt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:52.400540 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:52.400488 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7lt6\" (UniqueName: \"kubernetes.io/projected/5baf1c47-1d9f-4d68-93c8-7061a486469f-kube-api-access-d7lt6\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:52.400540 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:52.400512 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5baf1c47-1d9f-4d68-93c8-7061a486469f-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:52.400540 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:52.400522 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5baf1c47-1d9f-4d68-93c8-7061a486469f-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:52.667924 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:52.667839 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="84158686-4b22-4c87-ad4b-19a969a156fd" containerName="tempo-gateway" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:29:52.710042 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:52.710011 5274 generic.go:356] "Generic (PLEG): container finished" podID="84158686-4b22-4c87-ad4b-19a969a156fd" containerID="1dc5f97d29697d0c0a37a7fa4fda29b5f9e51fd614176bb3d81dbf84ac755a43" exitCode=0 Oct 31 00:29:52.710042 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:52.710034 5274 generic.go:356] "Generic (PLEG): container finished" podID="84158686-4b22-4c87-ad4b-19a969a156fd" containerID="f8f895b1e4d7c4cfba87d44a55e8443931cb4db38bd3558676817955dadf0ba6" exitCode=0 Oct 31 00:29:52.710227 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:52.710085 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"84158686-4b22-4c87-ad4b-19a969a156fd","Type":"ContainerDied","Data":"1dc5f97d29697d0c0a37a7fa4fda29b5f9e51fd614176bb3d81dbf84ac755a43"} Oct 31 00:29:52.710227 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:52.710133 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"84158686-4b22-4c87-ad4b-19a969a156fd","Type":"ContainerDied","Data":"f8f895b1e4d7c4cfba87d44a55e8443931cb4db38bd3558676817955dadf0ba6"} Oct 31 00:29:52.711280 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:52.711257 5274 generic.go:356] "Generic (PLEG): container finished" podID="5baf1c47-1d9f-4d68-93c8-7061a486469f" containerID="4d1372b886b8cee85033d37931df93f23f9fd0f323a7549aa134701c990479c3" exitCode=0 Oct 31 00:29:52.711400 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:52.711310 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-45gq9" Oct 31 00:29:52.711400 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:52.711341 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-45gq9" event={"ID":"5baf1c47-1d9f-4d68-93c8-7061a486469f","Type":"ContainerDied","Data":"4d1372b886b8cee85033d37931df93f23f9fd0f323a7549aa134701c990479c3"} Oct 31 00:29:52.711400 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:52.711394 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-45gq9" event={"ID":"5baf1c47-1d9f-4d68-93c8-7061a486469f","Type":"ContainerDied","Data":"84f17ae1db3fa7d3f4ee8b16d07f4460a481cc85f51722374802b3c46fa00c97"} Oct 31 00:29:52.711565 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:52.711424 5274 scope.go:122] "RemoveContainer" containerID="4d1372b886b8cee85033d37931df93f23f9fd0f323a7549aa134701c990479c3" Oct 31 00:29:52.719800 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:52.719781 5274 scope.go:122] "RemoveContainer" containerID="4d1372b886b8cee85033d37931df93f23f9fd0f323a7549aa134701c990479c3" Oct 31 00:29:52.720073 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:29:52.720053 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d1372b886b8cee85033d37931df93f23f9fd0f323a7549aa134701c990479c3\": container with ID starting with 4d1372b886b8cee85033d37931df93f23f9fd0f323a7549aa134701c990479c3 not found: ID does not exist" containerID="4d1372b886b8cee85033d37931df93f23f9fd0f323a7549aa134701c990479c3" Oct 31 00:29:52.720152 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:52.720082 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d1372b886b8cee85033d37931df93f23f9fd0f323a7549aa134701c990479c3"} err="failed to get container status \"4d1372b886b8cee85033d37931df93f23f9fd0f323a7549aa134701c990479c3\": rpc error: code = NotFound desc = could not find container \"4d1372b886b8cee85033d37931df93f23f9fd0f323a7549aa134701c990479c3\": container with ID starting with 4d1372b886b8cee85033d37931df93f23f9fd0f323a7549aa134701c990479c3 not found: ID does not exist" Oct 31 00:29:52.735232 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:52.735212 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-45gq9"] Oct 31 00:29:52.737122 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:52.737102 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-45gq9"] Oct 31 00:29:53.404983 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.404940 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:53.481526 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.481497 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5baf1c47-1d9f-4d68-93c8-7061a486469f" path="/var/lib/kubelet/pods/5baf1c47-1d9f-4d68-93c8-7061a486469f/volumes" Oct 31 00:29:53.510617 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.510566 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84158686-4b22-4c87-ad4b-19a969a156fd-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/84158686-4b22-4c87-ad4b-19a969a156fd-gateway-rbac\") pod \"84158686-4b22-4c87-ad4b-19a969a156fd\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") " Oct 31 00:29:53.510617 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.510607 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84158686-4b22-4c87-ad4b-19a969a156fd-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/84158686-4b22-4c87-ad4b-19a969a156fd-gateway-tenants\") pod \"84158686-4b22-4c87-ad4b-19a969a156fd\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") " Oct 31 00:29:53.510746 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.510640 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84158686-4b22-4c87-ad4b-19a969a156fd-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84158686-4b22-4c87-ad4b-19a969a156fd-tempo-data-science-tempomonolithic-ca-bundle\") pod \"84158686-4b22-4c87-ad4b-19a969a156fd\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") " Oct 31 00:29:53.510746 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.510707 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84158686-4b22-4c87-ad4b-19a969a156fd-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/84158686-4b22-4c87-ad4b-19a969a156fd-tempo-conf\") pod \"84158686-4b22-4c87-ad4b-19a969a156fd\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") " Oct 31 00:29:53.510836 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.510804 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"84158686-4b22-4c87-ad4b-19a969a156fd\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") " Oct 31 00:29:53.510892 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.510861 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84158686-4b22-4c87-ad4b-19a969a156fd-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/84158686-4b22-4c87-ad4b-19a969a156fd-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"84158686-4b22-4c87-ad4b-19a969a156fd\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") " Oct 31 00:29:53.510944 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.510901 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84158686-4b22-4c87-ad4b-19a969a156fd-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/84158686-4b22-4c87-ad4b-19a969a156fd-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"84158686-4b22-4c87-ad4b-19a969a156fd\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") " Oct 31 00:29:53.510944 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.510932 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84158686-4b22-4c87-ad4b-19a969a156fd-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/84158686-4b22-4c87-ad4b-19a969a156fd-tempo-data-science-tempomonolithic-mtls\") pod \"84158686-4b22-4c87-ad4b-19a969a156fd\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") " Oct 31 00:29:53.511086 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.510942 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84158686-4b22-4c87-ad4b-19a969a156fd-gateway-rbac" pod "84158686-4b22-4c87-ad4b-19a969a156fd" (UID: "84158686-4b22-4c87-ad4b-19a969a156fd"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:53.511086 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.510990 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84158686-4b22-4c87-ad4b-19a969a156fd-kube-api-access-q75wt\" (UniqueName: \"kubernetes.io/projected/84158686-4b22-4c87-ad4b-19a969a156fd-kube-api-access-q75wt\") pod \"84158686-4b22-4c87-ad4b-19a969a156fd\" (UID: \"84158686-4b22-4c87-ad4b-19a969a156fd\") " Oct 31 00:29:53.511086 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.511064 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84158686-4b22-4c87-ad4b-19a969a156fd-tempo-data-science-tempomonolithic-ca-bundle" pod "84158686-4b22-4c87-ad4b-19a969a156fd" (UID: "84158686-4b22-4c87-ad4b-19a969a156fd"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:53.511273 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.511255 5274 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/84158686-4b22-4c87-ad4b-19a969a156fd-gateway-rbac\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:53.511330 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.511271 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84158686-4b22-4c87-ad4b-19a969a156fd-tempo-conf" pod "84158686-4b22-4c87-ad4b-19a969a156fd" (UID: "84158686-4b22-4c87-ad4b-19a969a156fd"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:53.511330 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.511280 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84158686-4b22-4c87-ad4b-19a969a156fd-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:53.511433 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.511378 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84158686-4b22-4c87-ad4b-19a969a156fd-tempo-data-science-tempomonolithic-serving-cabundle" pod "84158686-4b22-4c87-ad4b-19a969a156fd" (UID: "84158686-4b22-4c87-ad4b-19a969a156fd"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:53.513296 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.513274 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84158686-4b22-4c87-ad4b-19a969a156fd-tempo-data-science-tempomonolithic-mtls" pod "84158686-4b22-4c87-ad4b-19a969a156fd" (UID: "84158686-4b22-4c87-ad4b-19a969a156fd"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:53.513423 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.513313 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84158686-4b22-4c87-ad4b-19a969a156fd-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "84158686-4b22-4c87-ad4b-19a969a156fd" (UID: "84158686-4b22-4c87-ad4b-19a969a156fd"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:53.513503 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.513484 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84158686-4b22-4c87-ad4b-19a969a156fd-gateway-tenants" pod "84158686-4b22-4c87-ad4b-19a969a156fd" (UID: "84158686-4b22-4c87-ad4b-19a969a156fd"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:53.513579 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.513561 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84158686-4b22-4c87-ad4b-19a969a156fd-kube-api-access-q75wt" pod "84158686-4b22-4c87-ad4b-19a969a156fd" (UID: "84158686-4b22-4c87-ad4b-19a969a156fd"). InnerVolumeSpecName "kube-api-access-q75wt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:53.513855 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.513837 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062" pod "84158686-4b22-4c87-ad4b-19a969a156fd" (UID: "84158686-4b22-4c87-ad4b-19a969a156fd"). InnerVolumeSpecName "pvc-4491ac70-4597-43a7-a792-f0813314e433". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:29:53.611993 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.611966 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/84158686-4b22-4c87-ad4b-19a969a156fd-tempo-conf\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:53.612103 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.612012 5274 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" " Oct 31 00:29:53.612103 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.612030 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/84158686-4b22-4c87-ad4b-19a969a156fd-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:53.612103 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.612046 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/84158686-4b22-4c87-ad4b-19a969a156fd-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:53.612103 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.612064 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/84158686-4b22-4c87-ad4b-19a969a156fd-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:53.612103 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.612079 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q75wt\" (UniqueName: \"kubernetes.io/projected/84158686-4b22-4c87-ad4b-19a969a156fd-kube-api-access-q75wt\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:53.612103 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.612092 5274 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/84158686-4b22-4c87-ad4b-19a969a156fd-gateway-tenants\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:53.620356 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.620340 5274 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-4491ac70-4597-43a7-a792-f0813314e433" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062") on node "ip-10-0-129-70.us-east-2.compute.internal" Oct 31 00:29:53.712616 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.712588 5274 reconciler_common.go:299] "Volume detached for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"csi-9ce3c80b5d5deac9dfdb953db4fb017080b654a85db76a2937e67cd541d89057\"" Oct 31 00:29:53.718737 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.718716 5274 generic.go:356] "Generic (PLEG): container finished" podID="84158686-4b22-4c87-ad4b-19a969a156fd" containerID="12746428d807a4221d75dc24ceae9d336797cee9ed6a9e56f8780aa736fdac27" exitCode=0 Oct 31 00:29:53.718834 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.718767 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:53.718834 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.718801 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"84158686-4b22-4c87-ad4b-19a969a156fd","Type":"ContainerDied","Data":"12746428d807a4221d75dc24ceae9d336797cee9ed6a9e56f8780aa736fdac27"} Oct 31 00:29:53.718920 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.718868 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"84158686-4b22-4c87-ad4b-19a969a156fd","Type":"ContainerDied","Data":"33ad5ae8d283f8d7dd970978f162a6f63d667d3a808bce7d309ae4efeb7bbee8"} Oct 31 00:29:53.718920 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.718903 5274 scope.go:122] "RemoveContainer" containerID="1dc5f97d29697d0c0a37a7fa4fda29b5f9e51fd614176bb3d81dbf84ac755a43" Oct 31 00:29:53.727162 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.727148 5274 scope.go:122] "RemoveContainer" containerID="12746428d807a4221d75dc24ceae9d336797cee9ed6a9e56f8780aa736fdac27" Oct 31 00:29:53.734806 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.734794 5274 scope.go:122] "RemoveContainer" containerID="f8f895b1e4d7c4cfba87d44a55e8443931cb4db38bd3558676817955dadf0ba6" Oct 31 00:29:53.741133 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.741112 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:29:53.742750 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.742731 5274 scope.go:122] "RemoveContainer" containerID="1dc5f97d29697d0c0a37a7fa4fda29b5f9e51fd614176bb3d81dbf84ac755a43" Oct 31 00:29:53.743030 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:29:53.743015 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dc5f97d29697d0c0a37a7fa4fda29b5f9e51fd614176bb3d81dbf84ac755a43\": container with ID starting with 1dc5f97d29697d0c0a37a7fa4fda29b5f9e51fd614176bb3d81dbf84ac755a43 not found: ID does not exist" containerID="1dc5f97d29697d0c0a37a7fa4fda29b5f9e51fd614176bb3d81dbf84ac755a43" Oct 31 00:29:53.743085 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.743036 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dc5f97d29697d0c0a37a7fa4fda29b5f9e51fd614176bb3d81dbf84ac755a43"} err="failed to get container status \"1dc5f97d29697d0c0a37a7fa4fda29b5f9e51fd614176bb3d81dbf84ac755a43\": rpc error: code = NotFound desc = could not find container \"1dc5f97d29697d0c0a37a7fa4fda29b5f9e51fd614176bb3d81dbf84ac755a43\": container with ID starting with 1dc5f97d29697d0c0a37a7fa4fda29b5f9e51fd614176bb3d81dbf84ac755a43 not found: ID does not exist" Oct 31 00:29:53.743085 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.743051 5274 scope.go:122] "RemoveContainer" containerID="12746428d807a4221d75dc24ceae9d336797cee9ed6a9e56f8780aa736fdac27" Oct 31 00:29:53.743296 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:29:53.743277 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12746428d807a4221d75dc24ceae9d336797cee9ed6a9e56f8780aa736fdac27\": container with ID starting with 12746428d807a4221d75dc24ceae9d336797cee9ed6a9e56f8780aa736fdac27 not found: ID does not exist" containerID="12746428d807a4221d75dc24ceae9d336797cee9ed6a9e56f8780aa736fdac27" Oct 31 00:29:53.743332 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.743301 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12746428d807a4221d75dc24ceae9d336797cee9ed6a9e56f8780aa736fdac27"} err="failed to get container status \"12746428d807a4221d75dc24ceae9d336797cee9ed6a9e56f8780aa736fdac27\": rpc error: code = NotFound desc = could not find container \"12746428d807a4221d75dc24ceae9d336797cee9ed6a9e56f8780aa736fdac27\": container with ID starting with 12746428d807a4221d75dc24ceae9d336797cee9ed6a9e56f8780aa736fdac27 not found: ID does not exist" Oct 31 00:29:53.743332 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.743315 5274 scope.go:122] "RemoveContainer" containerID="f8f895b1e4d7c4cfba87d44a55e8443931cb4db38bd3558676817955dadf0ba6" Oct 31 00:29:53.743539 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:29:53.743522 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8f895b1e4d7c4cfba87d44a55e8443931cb4db38bd3558676817955dadf0ba6\": container with ID starting with f8f895b1e4d7c4cfba87d44a55e8443931cb4db38bd3558676817955dadf0ba6 not found: ID does not exist" containerID="f8f895b1e4d7c4cfba87d44a55e8443931cb4db38bd3558676817955dadf0ba6" Oct 31 00:29:53.743580 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.743543 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f895b1e4d7c4cfba87d44a55e8443931cb4db38bd3558676817955dadf0ba6"} err="failed to get container status \"f8f895b1e4d7c4cfba87d44a55e8443931cb4db38bd3558676817955dadf0ba6\": rpc error: code = NotFound desc = could not find container \"f8f895b1e4d7c4cfba87d44a55e8443931cb4db38bd3558676817955dadf0ba6\": container with ID starting with f8f895b1e4d7c4cfba87d44a55e8443931cb4db38bd3558676817955dadf0ba6 not found: ID does not exist" Oct 31 00:29:53.747780 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:53.747746 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:29:54.716365 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:54.716335 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-d64sg"] Oct 31 00:29:54.716726 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:54.716652 5274 cpu_manager.go:414] "containerMap: removing container" podUID="84158686-4b22-4c87-ad4b-19a969a156fd" containerName="tempo-gateway" Oct 31 00:29:54.716726 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:54.716662 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84158686-4b22-4c87-ad4b-19a969a156fd" containerName="tempo-gateway" Oct 31 00:29:54.716726 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:54.716671 5274 cpu_manager.go:414] "containerMap: removing container" podUID="84158686-4b22-4c87-ad4b-19a969a156fd" containerName="tempo-gateway-opa" Oct 31 00:29:54.716726 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:54.716676 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84158686-4b22-4c87-ad4b-19a969a156fd" containerName="tempo-gateway-opa" Oct 31 00:29:54.716726 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:54.716686 5274 cpu_manager.go:414] "containerMap: removing container" podUID="84158686-4b22-4c87-ad4b-19a969a156fd" containerName="tempo" Oct 31 00:29:54.716726 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:54.716691 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84158686-4b22-4c87-ad4b-19a969a156fd" containerName="tempo" Oct 31 00:29:54.716726 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:54.716706 5274 cpu_manager.go:414] "containerMap: removing container" podUID="5baf1c47-1d9f-4d68-93c8-7061a486469f" containerName="korrel8r" Oct 31 00:29:54.716726 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:54.716711 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5baf1c47-1d9f-4d68-93c8-7061a486469f" containerName="korrel8r" Oct 31 00:29:54.716976 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:54.716762 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="84158686-4b22-4c87-ad4b-19a969a156fd" containerName="tempo-gateway" Oct 31 00:29:54.716976 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:54.716771 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="84158686-4b22-4c87-ad4b-19a969a156fd" containerName="tempo-gateway-opa" Oct 31 00:29:54.716976 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:54.716779 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="84158686-4b22-4c87-ad4b-19a969a156fd" containerName="tempo" Oct 31 00:29:54.716976 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:54.716786 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="5baf1c47-1d9f-4d68-93c8-7061a486469f" containerName="korrel8r" Oct 31 00:29:54.721298 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:54.721277 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-d64sg" Oct 31 00:29:54.728423 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:54.728376 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-dzj8s\"" Oct 31 00:29:54.728579 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:54.728442 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:29:54.728669 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:54.728615 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 31 00:29:54.728669 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:54.728633 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 31 00:29:54.729080 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:54.728846 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-d64sg"] Oct 31 00:29:54.729195 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:54.729173 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:29:54.820010 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:54.819983 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/8bd571b2-8f71-4546-a340-aabb2546c2bc-tls\") pod \"korrel8r-6644769df5-d64sg\" (UID: \"8bd571b2-8f71-4546-a340-aabb2546c2bc\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-d64sg" Oct 31 00:29:54.820148 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:54.820016 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8bd571b2-8f71-4546-a340-aabb2546c2bc-config\") pod \"korrel8r-6644769df5-d64sg\" (UID: \"8bd571b2-8f71-4546-a340-aabb2546c2bc\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-d64sg" Oct 31 00:29:54.820148 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:54.820105 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72fb5\" (UniqueName: \"kubernetes.io/projected/8bd571b2-8f71-4546-a340-aabb2546c2bc-kube-api-access-72fb5\") pod \"korrel8r-6644769df5-d64sg\" (UID: \"8bd571b2-8f71-4546-a340-aabb2546c2bc\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-d64sg" Oct 31 00:29:54.921372 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:54.921339 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/8bd571b2-8f71-4546-a340-aabb2546c2bc-tls\") pod \"korrel8r-6644769df5-d64sg\" (UID: \"8bd571b2-8f71-4546-a340-aabb2546c2bc\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-d64sg" Oct 31 00:29:54.921372 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:54.921373 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8bd571b2-8f71-4546-a340-aabb2546c2bc-config\") pod \"korrel8r-6644769df5-d64sg\" (UID: \"8bd571b2-8f71-4546-a340-aabb2546c2bc\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-d64sg" Oct 31 00:29:54.921547 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:54.921392 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72fb5\" (UniqueName: \"kubernetes.io/projected/8bd571b2-8f71-4546-a340-aabb2546c2bc-kube-api-access-72fb5\") pod \"korrel8r-6644769df5-d64sg\" (UID: \"8bd571b2-8f71-4546-a340-aabb2546c2bc\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-d64sg" Oct 31 00:29:54.922170 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:54.922147 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8bd571b2-8f71-4546-a340-aabb2546c2bc-config\") pod \"korrel8r-6644769df5-d64sg\" (UID: \"8bd571b2-8f71-4546-a340-aabb2546c2bc\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-d64sg" Oct 31 00:29:54.924614 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:54.924590 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/8bd571b2-8f71-4546-a340-aabb2546c2bc-tls\") pod \"korrel8r-6644769df5-d64sg\" (UID: \"8bd571b2-8f71-4546-a340-aabb2546c2bc\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-d64sg" Oct 31 00:29:54.934337 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:54.934311 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72fb5\" (UniqueName: \"kubernetes.io/projected/8bd571b2-8f71-4546-a340-aabb2546c2bc-kube-api-access-72fb5\") pod \"korrel8r-6644769df5-d64sg\" (UID: \"8bd571b2-8f71-4546-a340-aabb2546c2bc\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-d64sg" Oct 31 00:29:55.035987 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.035885 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-d64sg" Oct 31 00:29:55.319371 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.319346 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-d64sg"] Oct 31 00:29:55.321884 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:29:55.321857 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd571b2_8f71_4546_a340_aabb2546c2bc.slice/crio-09b1ff4081a537b25c1b462e0cc9010d4fa249887af4d59d91e664a131a7474e WatchSource:0}: Error finding container 09b1ff4081a537b25c1b462e0cc9010d4fa249887af4d59d91e664a131a7474e: Status 404 returned error can't find the container with id 09b1ff4081a537b25c1b462e0cc9010d4fa249887af4d59d91e664a131a7474e Oct 31 00:29:55.445703 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.445676 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:29:55.449456 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.449437 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:55.453330 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.453308 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\"" Oct 31 00:29:55.453592 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.453568 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\"" Oct 31 00:29:55.453592 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.453569 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\"" Oct 31 00:29:55.453741 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.453568 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 31 00:29:55.453741 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.453624 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 31 00:29:55.454184 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.454164 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\"" Oct 31 00:29:55.454294 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.454211 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\"" Oct 31 00:29:55.454294 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.454211 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-2tqfv\"" Oct 31 00:29:55.465737 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.465710 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:29:55.483561 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.483480 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84158686-4b22-4c87-ad4b-19a969a156fd" path="/var/lib/kubelet/pods/84158686-4b22-4c87-ad4b-19a969a156fd/volumes" Oct 31 00:29:55.626640 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.626611 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/f60809b3-41dc-42de-a403-ab2c96a808ee-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:55.626781 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.626661 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f60809b3-41dc-42de-a403-ab2c96a808ee-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:55.626781 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.626758 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/f60809b3-41dc-42de-a403-ab2c96a808ee-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:55.626870 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.626804 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/f60809b3-41dc-42de-a403-ab2c96a808ee-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:55.626919 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.626901 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/f60809b3-41dc-42de-a403-ab2c96a808ee-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:55.626974 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.626940 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/f60809b3-41dc-42de-a403-ab2c96a808ee-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:55.627016 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.626999 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f60809b3-41dc-42de-a403-ab2c96a808ee-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:55.627047 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.627021 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2jrv\" (UniqueName: \"kubernetes.io/projected/f60809b3-41dc-42de-a403-ab2c96a808ee-kube-api-access-g2jrv\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:55.728036 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.728007 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f60809b3-41dc-42de-a403-ab2c96a808ee-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:55.728491 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.728078 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/f60809b3-41dc-42de-a403-ab2c96a808ee-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:55.728491 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.728108 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/f60809b3-41dc-42de-a403-ab2c96a808ee-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:55.728491 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.728160 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/f60809b3-41dc-42de-a403-ab2c96a808ee-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:55.728491 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.728203 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/f60809b3-41dc-42de-a403-ab2c96a808ee-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:55.728491 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.728234 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f60809b3-41dc-42de-a403-ab2c96a808ee-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:55.728491 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.728264 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2jrv\" (UniqueName: \"kubernetes.io/projected/f60809b3-41dc-42de-a403-ab2c96a808ee-kube-api-access-g2jrv\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:55.728491 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.728296 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/f60809b3-41dc-42de-a403-ab2c96a808ee-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:55.728865 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.728832 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f60809b3-41dc-42de-a403-ab2c96a808ee-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:55.728926 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.728887 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/f60809b3-41dc-42de-a403-ab2c96a808ee-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:55.728926 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.728895 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/f60809b3-41dc-42de-a403-ab2c96a808ee-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:55.729431 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.729404 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f60809b3-41dc-42de-a403-ab2c96a808ee-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:55.730825 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.730799 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/f60809b3-41dc-42de-a403-ab2c96a808ee-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:55.730926 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.730851 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/f60809b3-41dc-42de-a403-ab2c96a808ee-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:55.730980 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.730923 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/f60809b3-41dc-42de-a403-ab2c96a808ee-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:55.732907 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.732884 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-d64sg" event={"ID":"8bd571b2-8f71-4546-a340-aabb2546c2bc","Type":"ContainerStarted","Data":"afd8ce3aea3e91a4d643898803e037d51ee4679fd3131ef3f9b673a142ae730f"} Oct 31 00:29:55.733049 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.732936 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-d64sg" event={"ID":"8bd571b2-8f71-4546-a340-aabb2546c2bc","Type":"ContainerStarted","Data":"09b1ff4081a537b25c1b462e0cc9010d4fa249887af4d59d91e664a131a7474e"} Oct 31 00:29:55.733049 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.733005 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-d64sg" Oct 31 00:29:55.741178 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.741160 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2jrv\" (UniqueName: \"kubernetes.io/projected/f60809b3-41dc-42de-a403-ab2c96a808ee-kube-api-access-g2jrv\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:29:55.752308 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:29:55.752263 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-6644769df5-d64sg" podStartSLOduration=1.75225187 podStartE2EDuration="1.75225187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:55.750886665 +0000 UTC m=+1832.387705383" watchObservedRunningTime="2026-10-31 00:29:55.75225187 +0000 UTC m=+1832.389070585" Oct 31 00:30:01.372459 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:01.372427 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:30:01.381819 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:01.381794 5274 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:30:01.473793 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:01.473761 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:30:01.473925 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:01.473873 5274 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:30:01.477554 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:01.477534 5274 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") DevicePath \"csi-9ce3c80b5d5deac9dfdb953db4fb017080b654a85db76a2937e67cd541d89057\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:30:01.568296 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:01.568264 5274 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/151619720cb39d40781bf258206fb0be746b77e5f55879e189e4cf0d6e46dfb0/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:30:01.585769 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:01.585738 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:30:01.769803 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:01.769719 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:30:02.057605 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:02.057509 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:30:02.061209 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:30:02.061176 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf60809b3_41dc_42de_a403_ab2c96a808ee.slice/crio-1888a7e7472019009319953f2248bffc9426fcf3d4a89a9ff0ef36b312bcb4f3 WatchSource:0}: Error finding container 1888a7e7472019009319953f2248bffc9426fcf3d4a89a9ff0ef36b312bcb4f3: Status 404 returned error can't find the container with id 1888a7e7472019009319953f2248bffc9426fcf3d4a89a9ff0ef36b312bcb4f3 Oct 31 00:30:02.768745 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:02.768702 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"f60809b3-41dc-42de-a403-ab2c96a808ee","Type":"ContainerStarted","Data":"52f14e892ea3a6b87bf2f6bbf3fe7395be6ceb1021d198a40780b005e8634325"} Oct 31 00:30:02.769223 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:02.768768 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"f60809b3-41dc-42de-a403-ab2c96a808ee","Type":"ContainerStarted","Data":"736a9e432945feaf9d8e46deb5b9c6a37259b47d92dd94d3a2fac76a61a48f02"} Oct 31 00:30:02.769223 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:02.768794 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"f60809b3-41dc-42de-a403-ab2c96a808ee","Type":"ContainerStarted","Data":"64b716b51eb9d54ef00220616f115e5138d35723fc3ad1ebb5c8c59b2e8b1a9e"} Oct 31 00:30:02.769223 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:02.768818 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"f60809b3-41dc-42de-a403-ab2c96a808ee","Type":"ContainerStarted","Data":"1888a7e7472019009319953f2248bffc9426fcf3d4a89a9ff0ef36b312bcb4f3"} Oct 31 00:30:02.769223 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:02.768849 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:30:02.769223 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:02.769000 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:30:02.771656 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:02.771636 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:30:02.778417 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:02.778397 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:30:02.796498 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:02.796434 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=7.796399102 podStartE2EDuration="7.796399102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:02.794949537 +0000 UTC m=+1839.431768254" watchObservedRunningTime="2026-10-31 00:30:02.796399102 +0000 UTC m=+1839.433217819" Oct 31 00:30:03.774613 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:03.774581 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:30:06.740112 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:06.740080 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-d64sg" Oct 31 00:30:20.898637 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:30:20.898597 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:30:24.781511 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:24.781464 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="f60809b3-41dc-42de-a403-ab2c96a808ee" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:30:34.781133 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:34.781082 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="f60809b3-41dc-42de-a403-ab2c96a808ee" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:30:44.780941 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:44.780909 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:30:47.086449 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:47.086411 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-d64sg"] Oct 31 00:30:47.086856 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:47.086611 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-6644769df5-d64sg" podUID="8bd571b2-8f71-4546-a340-aabb2546c2bc" containerName="korrel8r" containerID="cri-o://afd8ce3aea3e91a4d643898803e037d51ee4679fd3131ef3f9b673a142ae730f" gracePeriod=30 Oct 31 00:30:47.340723 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:47.340652 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-d64sg" Oct 31 00:30:47.383933 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:47.383898 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:30:47.384337 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:47.384212 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="f60809b3-41dc-42de-a403-ab2c96a808ee" containerName="tempo" containerID="cri-o://64b716b51eb9d54ef00220616f115e5138d35723fc3ad1ebb5c8c59b2e8b1a9e" gracePeriod=30 Oct 31 00:30:47.384421 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:47.384385 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="f60809b3-41dc-42de-a403-ab2c96a808ee" containerName="tempo-gateway-opa" containerID="cri-o://52f14e892ea3a6b87bf2f6bbf3fe7395be6ceb1021d198a40780b005e8634325" gracePeriod=30 Oct 31 00:30:47.384508 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:47.384487 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="f60809b3-41dc-42de-a403-ab2c96a808ee" containerName="tempo-gateway" containerID="cri-o://736a9e432945feaf9d8e46deb5b9c6a37259b47d92dd94d3a2fac76a61a48f02" gracePeriod=30 Oct 31 00:30:47.441649 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:47.441619 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bd571b2-8f71-4546-a340-aabb2546c2bc-kube-api-access-72fb5\" (UniqueName: \"kubernetes.io/projected/8bd571b2-8f71-4546-a340-aabb2546c2bc-kube-api-access-72fb5\") pod \"8bd571b2-8f71-4546-a340-aabb2546c2bc\" (UID: \"8bd571b2-8f71-4546-a340-aabb2546c2bc\") " Oct 31 00:30:47.441785 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:47.441665 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bd571b2-8f71-4546-a340-aabb2546c2bc-config\" (UniqueName: \"kubernetes.io/configmap/8bd571b2-8f71-4546-a340-aabb2546c2bc-config\") pod \"8bd571b2-8f71-4546-a340-aabb2546c2bc\" (UID: \"8bd571b2-8f71-4546-a340-aabb2546c2bc\") " Oct 31 00:30:47.441785 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:47.441714 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bd571b2-8f71-4546-a340-aabb2546c2bc-tls\" (UniqueName: \"kubernetes.io/secret/8bd571b2-8f71-4546-a340-aabb2546c2bc-tls\") pod \"8bd571b2-8f71-4546-a340-aabb2546c2bc\" (UID: \"8bd571b2-8f71-4546-a340-aabb2546c2bc\") " Oct 31 00:30:47.442165 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:47.442132 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bd571b2-8f71-4546-a340-aabb2546c2bc-config" pod "8bd571b2-8f71-4546-a340-aabb2546c2bc" (UID: "8bd571b2-8f71-4546-a340-aabb2546c2bc"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:47.443765 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:47.443735 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bd571b2-8f71-4546-a340-aabb2546c2bc-kube-api-access-72fb5" pod "8bd571b2-8f71-4546-a340-aabb2546c2bc" (UID: "8bd571b2-8f71-4546-a340-aabb2546c2bc"). InnerVolumeSpecName "kube-api-access-72fb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:47.443920 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:47.443896 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bd571b2-8f71-4546-a340-aabb2546c2bc-tls" pod "8bd571b2-8f71-4546-a340-aabb2546c2bc" (UID: "8bd571b2-8f71-4546-a340-aabb2546c2bc"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:47.543176 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:47.543154 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72fb5\" (UniqueName: \"kubernetes.io/projected/8bd571b2-8f71-4546-a340-aabb2546c2bc-kube-api-access-72fb5\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:47.543274 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:47.543178 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8bd571b2-8f71-4546-a340-aabb2546c2bc-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:47.543274 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:47.543189 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/8bd571b2-8f71-4546-a340-aabb2546c2bc-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:47.769688 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:47.769644 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="f60809b3-41dc-42de-a403-ab2c96a808ee" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.134.0.71:8083/ready\": dial tcp 10.134.0.71:8083: connect: connection refused" Oct 31 00:30:47.773431 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:47.773401 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="f60809b3-41dc-42de-a403-ab2c96a808ee" containerName="tempo-gateway" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:30:47.993842 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:47.993810 5274 generic.go:356] "Generic (PLEG): container finished" podID="f60809b3-41dc-42de-a403-ab2c96a808ee" containerID="52f14e892ea3a6b87bf2f6bbf3fe7395be6ceb1021d198a40780b005e8634325" exitCode=0 Oct 31 00:30:47.993842 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:47.993840 5274 generic.go:356] "Generic (PLEG): container finished" podID="f60809b3-41dc-42de-a403-ab2c96a808ee" containerID="64b716b51eb9d54ef00220616f115e5138d35723fc3ad1ebb5c8c59b2e8b1a9e" exitCode=0 Oct 31 00:30:47.994087 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:47.993863 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"f60809b3-41dc-42de-a403-ab2c96a808ee","Type":"ContainerDied","Data":"52f14e892ea3a6b87bf2f6bbf3fe7395be6ceb1021d198a40780b005e8634325"} Oct 31 00:30:47.994087 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:47.993917 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"f60809b3-41dc-42de-a403-ab2c96a808ee","Type":"ContainerDied","Data":"64b716b51eb9d54ef00220616f115e5138d35723fc3ad1ebb5c8c59b2e8b1a9e"} Oct 31 00:30:47.995092 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:47.995075 5274 generic.go:356] "Generic (PLEG): container finished" podID="8bd571b2-8f71-4546-a340-aabb2546c2bc" containerID="afd8ce3aea3e91a4d643898803e037d51ee4679fd3131ef3f9b673a142ae730f" exitCode=0 Oct 31 00:30:47.995194 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:47.995102 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-d64sg" event={"ID":"8bd571b2-8f71-4546-a340-aabb2546c2bc","Type":"ContainerDied","Data":"afd8ce3aea3e91a4d643898803e037d51ee4679fd3131ef3f9b673a142ae730f"} Oct 31 00:30:47.995194 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:47.995146 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-d64sg" Oct 31 00:30:47.995194 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:47.995154 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-d64sg" event={"ID":"8bd571b2-8f71-4546-a340-aabb2546c2bc","Type":"ContainerDied","Data":"09b1ff4081a537b25c1b462e0cc9010d4fa249887af4d59d91e664a131a7474e"} Oct 31 00:30:47.995194 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:47.995182 5274 scope.go:122] "RemoveContainer" containerID="afd8ce3aea3e91a4d643898803e037d51ee4679fd3131ef3f9b673a142ae730f" Oct 31 00:30:48.003425 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:48.003398 5274 scope.go:122] "RemoveContainer" containerID="afd8ce3aea3e91a4d643898803e037d51ee4679fd3131ef3f9b673a142ae730f" Oct 31 00:30:48.003724 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:30:48.003702 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afd8ce3aea3e91a4d643898803e037d51ee4679fd3131ef3f9b673a142ae730f\": container with ID starting with afd8ce3aea3e91a4d643898803e037d51ee4679fd3131ef3f9b673a142ae730f not found: ID does not exist" containerID="afd8ce3aea3e91a4d643898803e037d51ee4679fd3131ef3f9b673a142ae730f" Oct 31 00:30:48.003778 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:48.003733 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd8ce3aea3e91a4d643898803e037d51ee4679fd3131ef3f9b673a142ae730f"} err="failed to get container status \"afd8ce3aea3e91a4d643898803e037d51ee4679fd3131ef3f9b673a142ae730f\": rpc error: code = NotFound desc = could not find container \"afd8ce3aea3e91a4d643898803e037d51ee4679fd3131ef3f9b673a142ae730f\": container with ID starting with afd8ce3aea3e91a4d643898803e037d51ee4679fd3131ef3f9b673a142ae730f not found: ID does not exist" Oct 31 00:30:48.013461 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:48.013442 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-d64sg"] Oct 31 00:30:48.018010 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:48.017990 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-d64sg"] Oct 31 00:30:48.635516 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:48.635492 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:30:48.753335 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:48.753267 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"f60809b3-41dc-42de-a403-ab2c96a808ee\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") " Oct 31 00:30:48.753335 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:48.753303 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f60809b3-41dc-42de-a403-ab2c96a808ee-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/f60809b3-41dc-42de-a403-ab2c96a808ee-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"f60809b3-41dc-42de-a403-ab2c96a808ee\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") " Oct 31 00:30:48.753552 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:48.753342 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f60809b3-41dc-42de-a403-ab2c96a808ee-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f60809b3-41dc-42de-a403-ab2c96a808ee-tempo-data-science-tempomonolithic-ca-bundle\") pod \"f60809b3-41dc-42de-a403-ab2c96a808ee\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") " Oct 31 00:30:48.753552 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:48.753369 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f60809b3-41dc-42de-a403-ab2c96a808ee-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/f60809b3-41dc-42de-a403-ab2c96a808ee-gateway-rbac\") pod \"f60809b3-41dc-42de-a403-ab2c96a808ee\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") " Oct 31 00:30:48.753552 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:48.753408 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f60809b3-41dc-42de-a403-ab2c96a808ee-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/f60809b3-41dc-42de-a403-ab2c96a808ee-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"f60809b3-41dc-42de-a403-ab2c96a808ee\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") " Oct 31 00:30:48.753552 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:48.753426 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f60809b3-41dc-42de-a403-ab2c96a808ee-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f60809b3-41dc-42de-a403-ab2c96a808ee-tempo-conf\") pod \"f60809b3-41dc-42de-a403-ab2c96a808ee\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") " Oct 31 00:30:48.753552 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:48.753449 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f60809b3-41dc-42de-a403-ab2c96a808ee-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/f60809b3-41dc-42de-a403-ab2c96a808ee-gateway-tenants\") pod \"f60809b3-41dc-42de-a403-ab2c96a808ee\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") " Oct 31 00:30:48.753552 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:48.753497 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f60809b3-41dc-42de-a403-ab2c96a808ee-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/f60809b3-41dc-42de-a403-ab2c96a808ee-tempo-data-science-tempomonolithic-mtls\") pod \"f60809b3-41dc-42de-a403-ab2c96a808ee\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") " Oct 31 00:30:48.753552 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:48.753523 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f60809b3-41dc-42de-a403-ab2c96a808ee-kube-api-access-g2jrv\" (UniqueName: \"kubernetes.io/projected/f60809b3-41dc-42de-a403-ab2c96a808ee-kube-api-access-g2jrv\") pod \"f60809b3-41dc-42de-a403-ab2c96a808ee\" (UID: \"f60809b3-41dc-42de-a403-ab2c96a808ee\") " Oct 31 00:30:48.754041 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:48.754013 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f60809b3-41dc-42de-a403-ab2c96a808ee-tempo-data-science-tempomonolithic-serving-cabundle" pod "f60809b3-41dc-42de-a403-ab2c96a808ee" (UID: "f60809b3-41dc-42de-a403-ab2c96a808ee"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:48.754246 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:48.754142 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f60809b3-41dc-42de-a403-ab2c96a808ee-tempo-data-science-tempomonolithic-ca-bundle" pod "f60809b3-41dc-42de-a403-ab2c96a808ee" (UID: "f60809b3-41dc-42de-a403-ab2c96a808ee"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:48.754246 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:48.754152 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f60809b3-41dc-42de-a403-ab2c96a808ee-gateway-rbac" pod "f60809b3-41dc-42de-a403-ab2c96a808ee" (UID: "f60809b3-41dc-42de-a403-ab2c96a808ee"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:48.754246 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:48.754180 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f60809b3-41dc-42de-a403-ab2c96a808ee-tempo-conf" pod "f60809b3-41dc-42de-a403-ab2c96a808ee" (UID: "f60809b3-41dc-42de-a403-ab2c96a808ee"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:48.755928 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:48.755900 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f60809b3-41dc-42de-a403-ab2c96a808ee-tempo-data-science-tempomonolithic-mtls" pod "f60809b3-41dc-42de-a403-ab2c96a808ee" (UID: "f60809b3-41dc-42de-a403-ab2c96a808ee"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:48.755928 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:48.755916 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f60809b3-41dc-42de-a403-ab2c96a808ee-kube-api-access-g2jrv" pod "f60809b3-41dc-42de-a403-ab2c96a808ee" (UID: "f60809b3-41dc-42de-a403-ab2c96a808ee"). InnerVolumeSpecName "kube-api-access-g2jrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:48.756129 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:48.756105 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f60809b3-41dc-42de-a403-ab2c96a808ee-gateway-tenants" pod "f60809b3-41dc-42de-a403-ab2c96a808ee" (UID: "f60809b3-41dc-42de-a403-ab2c96a808ee"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:48.756182 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:48.756135 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f60809b3-41dc-42de-a403-ab2c96a808ee-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "f60809b3-41dc-42de-a403-ab2c96a808ee" (UID: "f60809b3-41dc-42de-a403-ab2c96a808ee"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:48.756510 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:48.756493 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062" pod "f60809b3-41dc-42de-a403-ab2c96a808ee" (UID: "f60809b3-41dc-42de-a403-ab2c96a808ee"). InnerVolumeSpecName "pvc-4491ac70-4597-43a7-a792-f0813314e433". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:30:48.854557 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:48.854530 5274 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" " Oct 31 00:30:48.854557 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:48.854557 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/f60809b3-41dc-42de-a403-ab2c96a808ee-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:48.854712 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:48.854570 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f60809b3-41dc-42de-a403-ab2c96a808ee-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:48.854712 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:48.854579 5274 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/f60809b3-41dc-42de-a403-ab2c96a808ee-gateway-rbac\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:48.854712 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:48.854589 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/f60809b3-41dc-42de-a403-ab2c96a808ee-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:48.854712 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:48.854598 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f60809b3-41dc-42de-a403-ab2c96a808ee-tempo-conf\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:48.854712 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:48.854607 5274 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/f60809b3-41dc-42de-a403-ab2c96a808ee-gateway-tenants\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:48.854712 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:48.854616 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/f60809b3-41dc-42de-a403-ab2c96a808ee-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:48.854712 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:48.854625 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2jrv\" (UniqueName: \"kubernetes.io/projected/f60809b3-41dc-42de-a403-ab2c96a808ee-kube-api-access-g2jrv\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:48.863490 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:48.863474 5274 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-4491ac70-4597-43a7-a792-f0813314e433" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062") on node "ip-10-0-129-70.us-east-2.compute.internal" Oct 31 00:30:48.955286 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:48.955262 5274 reconciler_common.go:299] "Volume detached for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"csi-9ce3c80b5d5deac9dfdb953db4fb017080b654a85db76a2937e67cd541d89057\"" Oct 31 00:30:49.006398 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:49.006326 5274 generic.go:356] "Generic (PLEG): container finished" podID="f60809b3-41dc-42de-a403-ab2c96a808ee" containerID="736a9e432945feaf9d8e46deb5b9c6a37259b47d92dd94d3a2fac76a61a48f02" exitCode=0 Oct 31 00:30:49.006398 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:49.006369 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"f60809b3-41dc-42de-a403-ab2c96a808ee","Type":"ContainerDied","Data":"736a9e432945feaf9d8e46deb5b9c6a37259b47d92dd94d3a2fac76a61a48f02"} Oct 31 00:30:49.006398 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:49.006384 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:30:49.006573 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:49.006425 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"f60809b3-41dc-42de-a403-ab2c96a808ee","Type":"ContainerDied","Data":"1888a7e7472019009319953f2248bffc9426fcf3d4a89a9ff0ef36b312bcb4f3"} Oct 31 00:30:49.006573 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:49.006455 5274 scope.go:122] "RemoveContainer" containerID="52f14e892ea3a6b87bf2f6bbf3fe7395be6ceb1021d198a40780b005e8634325" Oct 31 00:30:49.015312 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:49.015294 5274 scope.go:122] "RemoveContainer" containerID="736a9e432945feaf9d8e46deb5b9c6a37259b47d92dd94d3a2fac76a61a48f02" Oct 31 00:30:49.023142 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:49.023124 5274 scope.go:122] "RemoveContainer" containerID="64b716b51eb9d54ef00220616f115e5138d35723fc3ad1ebb5c8c59b2e8b1a9e" Oct 31 00:30:49.030883 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:49.030855 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:30:49.031450 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:49.031424 5274 scope.go:122] "RemoveContainer" containerID="52f14e892ea3a6b87bf2f6bbf3fe7395be6ceb1021d198a40780b005e8634325" Oct 31 00:30:49.031694 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:30:49.031676 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52f14e892ea3a6b87bf2f6bbf3fe7395be6ceb1021d198a40780b005e8634325\": container with ID starting with 52f14e892ea3a6b87bf2f6bbf3fe7395be6ceb1021d198a40780b005e8634325 not found: ID does not exist" containerID="52f14e892ea3a6b87bf2f6bbf3fe7395be6ceb1021d198a40780b005e8634325" Oct 31 00:30:49.031738 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:49.031703 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52f14e892ea3a6b87bf2f6bbf3fe7395be6ceb1021d198a40780b005e8634325"} err="failed to get container status \"52f14e892ea3a6b87bf2f6bbf3fe7395be6ceb1021d198a40780b005e8634325\": rpc error: code = NotFound desc = could not find container \"52f14e892ea3a6b87bf2f6bbf3fe7395be6ceb1021d198a40780b005e8634325\": container with ID starting with 52f14e892ea3a6b87bf2f6bbf3fe7395be6ceb1021d198a40780b005e8634325 not found: ID does not exist" Oct 31 00:30:49.031738 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:49.031719 5274 scope.go:122] "RemoveContainer" containerID="736a9e432945feaf9d8e46deb5b9c6a37259b47d92dd94d3a2fac76a61a48f02" Oct 31 00:30:49.032142 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:30:49.032094 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"736a9e432945feaf9d8e46deb5b9c6a37259b47d92dd94d3a2fac76a61a48f02\": container with ID starting with 736a9e432945feaf9d8e46deb5b9c6a37259b47d92dd94d3a2fac76a61a48f02 not found: ID does not exist" containerID="736a9e432945feaf9d8e46deb5b9c6a37259b47d92dd94d3a2fac76a61a48f02" Oct 31 00:30:49.032142 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:49.032125 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"736a9e432945feaf9d8e46deb5b9c6a37259b47d92dd94d3a2fac76a61a48f02"} err="failed to get container status \"736a9e432945feaf9d8e46deb5b9c6a37259b47d92dd94d3a2fac76a61a48f02\": rpc error: code = NotFound desc = could not find container \"736a9e432945feaf9d8e46deb5b9c6a37259b47d92dd94d3a2fac76a61a48f02\": container with ID starting with 736a9e432945feaf9d8e46deb5b9c6a37259b47d92dd94d3a2fac76a61a48f02 not found: ID does not exist" Oct 31 00:30:49.032293 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:49.032146 5274 scope.go:122] "RemoveContainer" containerID="64b716b51eb9d54ef00220616f115e5138d35723fc3ad1ebb5c8c59b2e8b1a9e" Oct 31 00:30:49.032874 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:30:49.032727 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64b716b51eb9d54ef00220616f115e5138d35723fc3ad1ebb5c8c59b2e8b1a9e\": container with ID starting with 64b716b51eb9d54ef00220616f115e5138d35723fc3ad1ebb5c8c59b2e8b1a9e not found: ID does not exist" containerID="64b716b51eb9d54ef00220616f115e5138d35723fc3ad1ebb5c8c59b2e8b1a9e" Oct 31 00:30:49.032874 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:49.032765 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64b716b51eb9d54ef00220616f115e5138d35723fc3ad1ebb5c8c59b2e8b1a9e"} err="failed to get container status \"64b716b51eb9d54ef00220616f115e5138d35723fc3ad1ebb5c8c59b2e8b1a9e\": rpc error: code = NotFound desc = could not find container \"64b716b51eb9d54ef00220616f115e5138d35723fc3ad1ebb5c8c59b2e8b1a9e\": container with ID starting with 64b716b51eb9d54ef00220616f115e5138d35723fc3ad1ebb5c8c59b2e8b1a9e not found: ID does not exist" Oct 31 00:30:49.034244 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:49.034227 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:30:49.481268 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:49.481234 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bd571b2-8f71-4546-a340-aabb2546c2bc" path="/var/lib/kubelet/pods/8bd571b2-8f71-4546-a340-aabb2546c2bc/volumes" Oct 31 00:30:49.481642 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:30:49.481630 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f60809b3-41dc-42de-a403-ab2c96a808ee" path="/var/lib/kubelet/pods/f60809b3-41dc-42de-a403-ab2c96a808ee/volumes" Oct 31 00:30:50.899494 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:30:50.899459 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:31:20.898426 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:31:20.898353 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:31:26.844518 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:26.844488 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-wmwbh"] Oct 31 00:31:26.844972 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:26.844818 5274 cpu_manager.go:414] "containerMap: removing container" podUID="f60809b3-41dc-42de-a403-ab2c96a808ee" containerName="tempo" Oct 31 00:31:26.844972 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:26.844829 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f60809b3-41dc-42de-a403-ab2c96a808ee" containerName="tempo" Oct 31 00:31:26.844972 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:26.844847 5274 cpu_manager.go:414] "containerMap: removing container" podUID="8bd571b2-8f71-4546-a340-aabb2546c2bc" containerName="korrel8r" Oct 31 00:31:26.844972 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:26.844852 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bd571b2-8f71-4546-a340-aabb2546c2bc" containerName="korrel8r" Oct 31 00:31:26.844972 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:26.844858 5274 cpu_manager.go:414] "containerMap: removing container" podUID="f60809b3-41dc-42de-a403-ab2c96a808ee" containerName="tempo-gateway" Oct 31 00:31:26.844972 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:26.844863 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f60809b3-41dc-42de-a403-ab2c96a808ee" containerName="tempo-gateway" Oct 31 00:31:26.844972 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:26.844872 5274 cpu_manager.go:414] "containerMap: removing container" podUID="f60809b3-41dc-42de-a403-ab2c96a808ee" containerName="tempo-gateway-opa" Oct 31 00:31:26.844972 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:26.844876 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f60809b3-41dc-42de-a403-ab2c96a808ee" containerName="tempo-gateway-opa" Oct 31 00:31:26.844972 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:26.844941 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="f60809b3-41dc-42de-a403-ab2c96a808ee" containerName="tempo-gateway-opa" Oct 31 00:31:26.844972 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:26.844966 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="f60809b3-41dc-42de-a403-ab2c96a808ee" containerName="tempo" Oct 31 00:31:26.844972 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:26.844975 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bd571b2-8f71-4546-a340-aabb2546c2bc" containerName="korrel8r" Oct 31 00:31:26.845305 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:26.844980 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="f60809b3-41dc-42de-a403-ab2c96a808ee" containerName="tempo-gateway" Oct 31 00:31:26.849348 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:26.849331 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wmwbh" Oct 31 00:31:26.856442 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:26.856423 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 31 00:31:26.859523 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:26.859503 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:31:26.859624 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:26.859570 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:31:26.860846 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:26.860820 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-zmk4l\"" Oct 31 00:31:26.860967 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:26.860914 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 31 00:31:26.862323 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:26.862301 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-wmwbh"] Oct 31 00:31:26.943011 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:26.942989 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/591737eb-62c5-45cd-8e05-537ea893a810-config\") pod \"korrel8r-5d45699dc4-wmwbh\" (UID: \"591737eb-62c5-45cd-8e05-537ea893a810\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wmwbh" Oct 31 00:31:26.943135 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:26.943029 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f5rz\" (UniqueName: \"kubernetes.io/projected/591737eb-62c5-45cd-8e05-537ea893a810-kube-api-access-4f5rz\") pod \"korrel8r-5d45699dc4-wmwbh\" (UID: \"591737eb-62c5-45cd-8e05-537ea893a810\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wmwbh" Oct 31 00:31:26.943135 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:26.943055 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/591737eb-62c5-45cd-8e05-537ea893a810-tls\") pod \"korrel8r-5d45699dc4-wmwbh\" (UID: \"591737eb-62c5-45cd-8e05-537ea893a810\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wmwbh" Oct 31 00:31:27.043698 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:27.043673 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/591737eb-62c5-45cd-8e05-537ea893a810-config\") pod \"korrel8r-5d45699dc4-wmwbh\" (UID: \"591737eb-62c5-45cd-8e05-537ea893a810\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wmwbh" Oct 31 00:31:27.043845 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:27.043718 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f5rz\" (UniqueName: \"kubernetes.io/projected/591737eb-62c5-45cd-8e05-537ea893a810-kube-api-access-4f5rz\") pod \"korrel8r-5d45699dc4-wmwbh\" (UID: \"591737eb-62c5-45cd-8e05-537ea893a810\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wmwbh" Oct 31 00:31:27.043845 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:27.043743 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/591737eb-62c5-45cd-8e05-537ea893a810-tls\") pod \"korrel8r-5d45699dc4-wmwbh\" (UID: \"591737eb-62c5-45cd-8e05-537ea893a810\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wmwbh" Oct 31 00:31:27.044493 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:27.044468 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/591737eb-62c5-45cd-8e05-537ea893a810-config\") pod \"korrel8r-5d45699dc4-wmwbh\" (UID: \"591737eb-62c5-45cd-8e05-537ea893a810\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wmwbh" Oct 31 00:31:27.046151 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:27.046129 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/591737eb-62c5-45cd-8e05-537ea893a810-tls\") pod \"korrel8r-5d45699dc4-wmwbh\" (UID: \"591737eb-62c5-45cd-8e05-537ea893a810\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wmwbh" Oct 31 00:31:27.056420 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:27.056399 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f5rz\" (UniqueName: \"kubernetes.io/projected/591737eb-62c5-45cd-8e05-537ea893a810-kube-api-access-4f5rz\") pod \"korrel8r-5d45699dc4-wmwbh\" (UID: \"591737eb-62c5-45cd-8e05-537ea893a810\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wmwbh" Oct 31 00:31:27.143052 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:27.143016 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:31:27.147146 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:27.147127 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:31:27.153389 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:27.153350 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:31:27.162169 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:27.162149 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wmwbh" Oct 31 00:31:27.230579 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:27.230172 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:31:27.230579 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:27.230320 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-54f83bab\"" Oct 31 00:31:27.230579 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:27.230447 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:31:27.230579 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:27.230546 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rkjdd\"" Oct 31 00:31:27.245724 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:27.245553 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ad8f04e7-bb53-4453-9967-8bd1df042a74-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ad8f04e7-bb53-4453-9967-8bd1df042a74\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:31:27.245724 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:27.245594 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad8f04e7-bb53-4453-9967-8bd1df042a74-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ad8f04e7-bb53-4453-9967-8bd1df042a74\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:31:27.245724 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:27.245627 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad8f04e7-bb53-4453-9967-8bd1df042a74-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ad8f04e7-bb53-4453-9967-8bd1df042a74\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:31:27.245724 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:27.245644 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsxhw\" (UniqueName: \"kubernetes.io/projected/ad8f04e7-bb53-4453-9967-8bd1df042a74-kube-api-access-vsxhw\") pod \"data-science-collector-collector-1\" (UID: \"ad8f04e7-bb53-4453-9967-8bd1df042a74\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:31:27.346618 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:27.346586 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad8f04e7-bb53-4453-9967-8bd1df042a74-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ad8f04e7-bb53-4453-9967-8bd1df042a74\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:31:27.346770 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:27.346633 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad8f04e7-bb53-4453-9967-8bd1df042a74-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ad8f04e7-bb53-4453-9967-8bd1df042a74\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:31:27.346770 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:27.346660 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsxhw\" (UniqueName: \"kubernetes.io/projected/ad8f04e7-bb53-4453-9967-8bd1df042a74-kube-api-access-vsxhw\") pod \"data-science-collector-collector-1\" (UID: \"ad8f04e7-bb53-4453-9967-8bd1df042a74\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:31:27.346770 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:27.346717 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ad8f04e7-bb53-4453-9967-8bd1df042a74-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ad8f04e7-bb53-4453-9967-8bd1df042a74\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:31:27.347457 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:27.347430 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ad8f04e7-bb53-4453-9967-8bd1df042a74-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ad8f04e7-bb53-4453-9967-8bd1df042a74\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:31:27.349173 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:27.349151 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad8f04e7-bb53-4453-9967-8bd1df042a74-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ad8f04e7-bb53-4453-9967-8bd1df042a74\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:31:27.349263 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:27.349182 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad8f04e7-bb53-4453-9967-8bd1df042a74-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ad8f04e7-bb53-4453-9967-8bd1df042a74\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:31:27.361411 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:27.361390 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsxhw\" (UniqueName: \"kubernetes.io/projected/ad8f04e7-bb53-4453-9967-8bd1df042a74-kube-api-access-vsxhw\") pod \"data-science-collector-collector-1\" (UID: \"ad8f04e7-bb53-4453-9967-8bd1df042a74\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:31:27.440755 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:27.440732 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-wmwbh"] Oct 31 00:31:27.442528 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:31:27.442499 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod591737eb_62c5_45cd_8e05_537ea893a810.slice/crio-c65d84f462a19203f19b6b4db5469907d71b0a617893b6a9c4173a2d1db1462c WatchSource:0}: Error finding container c65d84f462a19203f19b6b4db5469907d71b0a617893b6a9c4173a2d1db1462c: Status 404 returned error can't find the container with id c65d84f462a19203f19b6b4db5469907d71b0a617893b6a9c4173a2d1db1462c Oct 31 00:31:27.464634 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:27.464611 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:31:27.742589 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:27.742566 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:31:27.745119 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:31:27.745092 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad8f04e7_bb53_4453_9967_8bd1df042a74.slice/crio-9623f202ad296301784e25da11b7f1bda80dbc12bc18dd127b4003278d07a27f WatchSource:0}: Error finding container 9623f202ad296301784e25da11b7f1bda80dbc12bc18dd127b4003278d07a27f: Status 404 returned error can't find the container with id 9623f202ad296301784e25da11b7f1bda80dbc12bc18dd127b4003278d07a27f Oct 31 00:31:27.747326 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:27.747290 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:31:27.747418 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:27.747351 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:31:28.199856 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:28.199824 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ad8f04e7-bb53-4453-9967-8bd1df042a74","Type":"ContainerStarted","Data":"adaa041afb52b1848236e86b521aa6841aa3a67114e3d36189a2102d8b732c13"} Oct 31 00:31:28.200330 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:28.199880 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ad8f04e7-bb53-4453-9967-8bd1df042a74","Type":"ContainerStarted","Data":"9623f202ad296301784e25da11b7f1bda80dbc12bc18dd127b4003278d07a27f"} Oct 31 00:31:28.201538 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:28.201501 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wmwbh" event={"ID":"591737eb-62c5-45cd-8e05-537ea893a810","Type":"ContainerStarted","Data":"8ad77c08c09cfae07f190503b14f868f44cb0437bd6539f1bcc5c0eb9b3b9ab9"} Oct 31 00:31:28.201648 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:28.201545 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wmwbh" event={"ID":"591737eb-62c5-45cd-8e05-537ea893a810","Type":"ContainerStarted","Data":"c65d84f462a19203f19b6b4db5469907d71b0a617893b6a9c4173a2d1db1462c"} Oct 31 00:31:28.201648 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:28.201576 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wmwbh" Oct 31 00:31:28.219642 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:28.219604 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.219592194 podStartE2EDuration="1.219592194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:31:28.218463665 +0000 UTC m=+1924.855282384" watchObservedRunningTime="2026-10-31 00:31:28.219592194 +0000 UTC m=+1924.856410909" Oct 31 00:31:28.237606 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:28.237565 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wmwbh" podStartSLOduration=2.237552485 podStartE2EDuration="2.237552485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:31:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:31:28.236272658 +0000 UTC m=+1924.873091377" watchObservedRunningTime="2026-10-31 00:31:28.237552485 +0000 UTC m=+1924.874371195" Oct 31 00:31:29.882895 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:29.882861 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-66476ccc6-t4v98"] Oct 31 00:31:29.886455 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:29.886429 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-66476ccc6-t4v98" Oct 31 00:31:29.892840 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:29.892813 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-dockercfg-5wl9c\"" Oct 31 00:31:29.900047 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:29.898302 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-ca-bundle\"" Oct 31 00:31:29.900047 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:29.898470 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-distributor-mtls\"" Oct 31 00:31:29.908786 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:29.904699 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack\"" Oct 31 00:31:29.908786 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:29.908208 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-66476ccc6-t4v98"] Oct 31 00:31:29.965570 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:29.965498 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5dece37-633f-4226-a0f0-f672aa15b1e6-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-66476ccc6-t4v98\" (UID: \"d5dece37-633f-4226-a0f0-f672aa15b1e6\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-66476ccc6-t4v98" Oct 31 00:31:29.965742 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:29.965650 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrc77\" (UniqueName: \"kubernetes.io/projected/d5dece37-633f-4226-a0f0-f672aa15b1e6-kube-api-access-zrc77\") pod \"tempo-data-science-tempostack-distributor-66476ccc6-t4v98\" (UID: \"d5dece37-633f-4226-a0f0-f672aa15b1e6\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-66476ccc6-t4v98" Oct 31 00:31:29.965805 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:29.965740 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/d5dece37-633f-4226-a0f0-f672aa15b1e6-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-66476ccc6-t4v98\" (UID: \"d5dece37-633f-4226-a0f0-f672aa15b1e6\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-66476ccc6-t4v98" Oct 31 00:31:29.965805 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:29.965776 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d5dece37-633f-4226-a0f0-f672aa15b1e6-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-66476ccc6-t4v98\" (UID: \"d5dece37-633f-4226-a0f0-f672aa15b1e6\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-66476ccc6-t4v98" Oct 31 00:31:29.965805 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:29.965796 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/d5dece37-633f-4226-a0f0-f672aa15b1e6-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-66476ccc6-t4v98\" (UID: \"d5dece37-633f-4226-a0f0-f672aa15b1e6\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-66476ccc6-t4v98" Oct 31 00:31:29.973449 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:29.973415 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2"] Oct 31 00:31:29.977611 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:29.977589 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2" Oct 31 00:31:29.990375 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:29.989758 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2"] Oct 31 00:31:29.993189 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:29.993163 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-query-frontend-mtls\"" Oct 31 00:31:29.993290 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:29.993267 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"s3-secret\"" Oct 31 00:31:30.066394 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:30.066366 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kd48\" (UniqueName: \"kubernetes.io/projected/08a395e2-f3b1-405f-bdc2-a324d281136f-kube-api-access-7kd48\") pod \"tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2\" (UID: \"08a395e2-f3b1-405f-bdc2-a324d281136f\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2" Oct 31 00:31:30.066394 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:30.066397 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/08a395e2-f3b1-405f-bdc2-a324d281136f-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2\" (UID: \"08a395e2-f3b1-405f-bdc2-a324d281136f\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2" Oct 31 00:31:30.066647 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:30.066443 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/d5dece37-633f-4226-a0f0-f672aa15b1e6-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-66476ccc6-t4v98\" (UID: \"d5dece37-633f-4226-a0f0-f672aa15b1e6\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-66476ccc6-t4v98" Oct 31 00:31:30.066710 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:30.066661 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d5dece37-633f-4226-a0f0-f672aa15b1e6-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-66476ccc6-t4v98\" (UID: \"d5dece37-633f-4226-a0f0-f672aa15b1e6\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-66476ccc6-t4v98" Oct 31 00:31:30.066710 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:30.066691 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/d5dece37-633f-4226-a0f0-f672aa15b1e6-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-66476ccc6-t4v98\" (UID: \"d5dece37-633f-4226-a0f0-f672aa15b1e6\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-66476ccc6-t4v98" Oct 31 00:31:30.066810 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:30.066721 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08a395e2-f3b1-405f-bdc2-a324d281136f-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2\" (UID: \"08a395e2-f3b1-405f-bdc2-a324d281136f\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2" Oct 31 00:31:30.066810 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:30.066754 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5dece37-633f-4226-a0f0-f672aa15b1e6-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-66476ccc6-t4v98\" (UID: \"d5dece37-633f-4226-a0f0-f672aa15b1e6\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-66476ccc6-t4v98" Oct 31 00:31:30.066810 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:30.066782 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/08a395e2-f3b1-405f-bdc2-a324d281136f-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2\" (UID: \"08a395e2-f3b1-405f-bdc2-a324d281136f\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2" Oct 31 00:31:30.066997 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:30.066903 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/08a395e2-f3b1-405f-bdc2-a324d281136f-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2\" (UID: \"08a395e2-f3b1-405f-bdc2-a324d281136f\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2" Oct 31 00:31:30.067099 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:30.067008 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrc77\" (UniqueName: \"kubernetes.io/projected/d5dece37-633f-4226-a0f0-f672aa15b1e6-kube-api-access-zrc77\") pod \"tempo-data-science-tempostack-distributor-66476ccc6-t4v98\" (UID: \"d5dece37-633f-4226-a0f0-f672aa15b1e6\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-66476ccc6-t4v98" Oct 31 00:31:30.067165 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:30.067131 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/d5dece37-633f-4226-a0f0-f672aa15b1e6-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-66476ccc6-t4v98\" (UID: \"d5dece37-633f-4226-a0f0-f672aa15b1e6\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-66476ccc6-t4v98" Oct 31 00:31:30.067562 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:30.067535 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5dece37-633f-4226-a0f0-f672aa15b1e6-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-66476ccc6-t4v98\" (UID: \"d5dece37-633f-4226-a0f0-f672aa15b1e6\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-66476ccc6-t4v98" Oct 31 00:31:30.067661 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:30.067617 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d5dece37-633f-4226-a0f0-f672aa15b1e6-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-66476ccc6-t4v98\" (UID: \"d5dece37-633f-4226-a0f0-f672aa15b1e6\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-66476ccc6-t4v98" Oct 31 00:31:30.068996 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:30.068948 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/d5dece37-633f-4226-a0f0-f672aa15b1e6-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-66476ccc6-t4v98\" (UID: \"d5dece37-633f-4226-a0f0-f672aa15b1e6\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-66476ccc6-t4v98" Oct 31 00:31:30.079782 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:30.079758 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrc77\" (UniqueName: \"kubernetes.io/projected/d5dece37-633f-4226-a0f0-f672aa15b1e6-kube-api-access-zrc77\") pod \"tempo-data-science-tempostack-distributor-66476ccc6-t4v98\" (UID: \"d5dece37-633f-4226-a0f0-f672aa15b1e6\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-66476ccc6-t4v98" Oct 31 00:31:30.168476 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:30.168419 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08a395e2-f3b1-405f-bdc2-a324d281136f-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2\" (UID: \"08a395e2-f3b1-405f-bdc2-a324d281136f\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2" Oct 31 00:31:30.168476 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:30.168452 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/08a395e2-f3b1-405f-bdc2-a324d281136f-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2\" (UID: \"08a395e2-f3b1-405f-bdc2-a324d281136f\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2" Oct 31 00:31:30.168476 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:30.168475 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/08a395e2-f3b1-405f-bdc2-a324d281136f-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2\" (UID: \"08a395e2-f3b1-405f-bdc2-a324d281136f\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2" Oct 31 00:31:30.168669 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:30.168512 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kd48\" (UniqueName: \"kubernetes.io/projected/08a395e2-f3b1-405f-bdc2-a324d281136f-kube-api-access-7kd48\") pod \"tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2\" (UID: \"08a395e2-f3b1-405f-bdc2-a324d281136f\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2" Oct 31 00:31:30.168669 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:30.168542 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/08a395e2-f3b1-405f-bdc2-a324d281136f-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2\" (UID: \"08a395e2-f3b1-405f-bdc2-a324d281136f\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2" Oct 31 00:31:30.168904 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:30.168888 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/08a395e2-f3b1-405f-bdc2-a324d281136f-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2\" (UID: \"08a395e2-f3b1-405f-bdc2-a324d281136f\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2" Oct 31 00:31:30.169148 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:30.169127 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08a395e2-f3b1-405f-bdc2-a324d281136f-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2\" (UID: \"08a395e2-f3b1-405f-bdc2-a324d281136f\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2" Oct 31 00:31:30.169362 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:30.169344 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/08a395e2-f3b1-405f-bdc2-a324d281136f-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2\" (UID: \"08a395e2-f3b1-405f-bdc2-a324d281136f\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2" Oct 31 00:31:30.170845 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:30.170823 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/08a395e2-f3b1-405f-bdc2-a324d281136f-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2\" (UID: \"08a395e2-f3b1-405f-bdc2-a324d281136f\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2" Oct 31 00:31:30.182772 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:30.182746 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kd48\" (UniqueName: \"kubernetes.io/projected/08a395e2-f3b1-405f-bdc2-a324d281136f-kube-api-access-7kd48\") pod \"tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2\" (UID: \"08a395e2-f3b1-405f-bdc2-a324d281136f\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2" Oct 31 00:31:30.213052 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:30.213032 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-66476ccc6-t4v98" Oct 31 00:31:30.293304 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:30.293273 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2" Oct 31 00:31:30.492813 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:30.492786 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-66476ccc6-t4v98"] Oct 31 00:31:30.494585 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:31:30.494561 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5dece37_633f_4226_a0f0_f672aa15b1e6.slice/crio-0db296a002751532dab5458752e05049b9ef89f9408fb659420da244084d023b WatchSource:0}: Error finding container 0db296a002751532dab5458752e05049b9ef89f9408fb659420da244084d023b: Status 404 returned error can't find the container with id 0db296a002751532dab5458752e05049b9ef89f9408fb659420da244084d023b Oct 31 00:31:30.570815 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:30.570791 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2"] Oct 31 00:31:30.572927 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:31:30.572903 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08a395e2_f3b1_405f_bdc2_a324d281136f.slice/crio-994f243ab1bf645aa6b28899cc8e1e30049a70953771ed1e21729eec234b752c WatchSource:0}: Error finding container 994f243ab1bf645aa6b28899cc8e1e30049a70953771ed1e21729eec234b752c: Status 404 returned error can't find the container with id 994f243ab1bf645aa6b28899cc8e1e30049a70953771ed1e21729eec234b752c Oct 31 00:31:31.218340 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:31.218301 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2" event={"ID":"08a395e2-f3b1-405f-bdc2-a324d281136f","Type":"ContainerStarted","Data":"362acdb4b1f0be1f0c6a672d608b5c21f4fc1fbf2c83bd7d84caa593aa4b0ccf"} Oct 31 00:31:31.218785 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:31.218452 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2" Oct 31 00:31:31.218785 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:31.218504 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2" event={"ID":"08a395e2-f3b1-405f-bdc2-a324d281136f","Type":"ContainerStarted","Data":"994f243ab1bf645aa6b28899cc8e1e30049a70953771ed1e21729eec234b752c"} Oct 31 00:31:31.219992 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:31.219948 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-66476ccc6-t4v98" event={"ID":"d5dece37-633f-4226-a0f0-f672aa15b1e6","Type":"ContainerStarted","Data":"928b9e1dee765cf1b987c36d65cd2cad9d6714d73bd5441d310d10788ebdf243"} Oct 31 00:31:31.220085 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:31.220013 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-66476ccc6-t4v98" event={"ID":"d5dece37-633f-4226-a0f0-f672aa15b1e6","Type":"ContainerStarted","Data":"0db296a002751532dab5458752e05049b9ef89f9408fb659420da244084d023b"} Oct 31 00:31:31.220085 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:31.220034 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-66476ccc6-t4v98" Oct 31 00:31:31.239854 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:31.239808 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2" podStartSLOduration=2.239796222 podStartE2EDuration="2.239796222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:31:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:31:31.237656633 +0000 UTC m=+1927.874475342" watchObservedRunningTime="2026-10-31 00:31:31.239796222 +0000 UTC m=+1927.876614938" Oct 31 00:31:31.258610 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:31.258568 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-66476ccc6-t4v98" podStartSLOduration=2.258556139 podStartE2EDuration="2.258556139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:31:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:31:31.256689642 +0000 UTC m=+1927.893508358" watchObservedRunningTime="2026-10-31 00:31:31.258556139 +0000 UTC m=+1927.895374854" Oct 31 00:31:32.944198 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:32.944165 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 31 00:31:32.947829 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:32.947811 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:31:32.952556 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:32.952529 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-ingester-mtls\"" Oct 31 00:31:32.960137 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:32.960113 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 31 00:31:32.992824 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:32.992799 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/5e6f500c-b181-456e-b957-4e5defa058f8-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"5e6f500c-b181-456e-b957-4e5defa058f8\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:31:32.992935 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:32.992829 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e6f500c-b181-456e-b957-4e5defa058f8-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"5e6f500c-b181-456e-b957-4e5defa058f8\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:31:32.992935 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:32.992867 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/5e6f500c-b181-456e-b957-4e5defa058f8-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"5e6f500c-b181-456e-b957-4e5defa058f8\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:31:32.993056 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:32.992974 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9447t\" (UniqueName: \"kubernetes.io/projected/5e6f500c-b181-456e-b957-4e5defa058f8-kube-api-access-9447t\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"5e6f500c-b181-456e-b957-4e5defa058f8\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:31:33.093772 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:33.093749 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/5e6f500c-b181-456e-b957-4e5defa058f8-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"5e6f500c-b181-456e-b957-4e5defa058f8\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:31:33.093889 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:33.093782 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e6f500c-b181-456e-b957-4e5defa058f8-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"5e6f500c-b181-456e-b957-4e5defa058f8\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:31:33.093889 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:33.093812 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/5e6f500c-b181-456e-b957-4e5defa058f8-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"5e6f500c-b181-456e-b957-4e5defa058f8\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:31:33.093889 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:33.093840 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9447t\" (UniqueName: \"kubernetes.io/projected/5e6f500c-b181-456e-b957-4e5defa058f8-kube-api-access-9447t\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"5e6f500c-b181-456e-b957-4e5defa058f8\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:31:33.094481 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:33.094461 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e6f500c-b181-456e-b957-4e5defa058f8-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"5e6f500c-b181-456e-b957-4e5defa058f8\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:31:33.094655 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:33.094636 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/5e6f500c-b181-456e-b957-4e5defa058f8-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"5e6f500c-b181-456e-b957-4e5defa058f8\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:31:33.096272 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:33.096226 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/5e6f500c-b181-456e-b957-4e5defa058f8-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"5e6f500c-b181-456e-b957-4e5defa058f8\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:31:33.106650 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:33.106627 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9447t\" (UniqueName: \"kubernetes.io/projected/5e6f500c-b181-456e-b957-4e5defa058f8-kube-api-access-9447t\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"5e6f500c-b181-456e-b957-4e5defa058f8\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:31:35.411005 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:35.410972 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-1a0c5ca2-2766-4db0-b17f-4a8f45925f49\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aac1093750eb10b6\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"5e6f500c-b181-456e-b957-4e5defa058f8\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:31:35.418001 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:35.417976 5274 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-1a0c5ca2-2766-4db0-b17f-4a8f45925f49\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aac1093750eb10b6\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"5e6f500c-b181-456e-b957-4e5defa058f8\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:31:35.511655 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:35.511624 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-1a0c5ca2-2766-4db0-b17f-4a8f45925f49\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aac1093750eb10b6\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"5e6f500c-b181-456e-b957-4e5defa058f8\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:31:35.511765 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:35.511749 5274 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-1a0c5ca2-2766-4db0-b17f-4a8f45925f49\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aac1093750eb10b6\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"5e6f500c-b181-456e-b957-4e5defa058f8\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:31:35.515286 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:35.515266 5274 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-1a0c5ca2-2766-4db0-b17f-4a8f45925f49\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aac1093750eb10b6\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"5e6f500c-b181-456e-b957-4e5defa058f8\") DevicePath \"csi-8a41e48b52e68bf1c119110f180532301f2a775d1f1f46a0f1f5e5aa4e023bca\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:31:35.739332 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:35.739296 5274 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-1a0c5ca2-2766-4db0-b17f-4a8f45925f49\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aac1093750eb10b6\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"5e6f500c-b181-456e-b957-4e5defa058f8\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/1018b4fc58a16f6eda6b1e7c04ae4351db82acf0495116e38742f3ca07150739/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:31:35.752414 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:35.752392 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-1a0c5ca2-2766-4db0-b17f-4a8f45925f49\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aac1093750eb10b6\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"5e6f500c-b181-456e-b957-4e5defa058f8\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:31:35.961627 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:35.961599 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:31:36.237658 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:36.237621 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 31 00:31:37.254300 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:37.254263 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"5e6f500c-b181-456e-b957-4e5defa058f8","Type":"ContainerStarted","Data":"d3e0b60f787ee6ad1dbfbde5cd69e467810c6191b51554433cbbc9d757c8fb94"} Oct 31 00:31:37.254702 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:37.254315 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"5e6f500c-b181-456e-b957-4e5defa058f8","Type":"ContainerStarted","Data":"a8e9fa461e879c4dad683f10b466d82e79f40ef656c570b12d19600fe4998e74"} Oct 31 00:31:37.254702 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:37.254411 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:31:37.277948 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:37.277890 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podStartSLOduration=8.277878232 podStartE2EDuration="8.277878232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:31:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:31:37.276022488 +0000 UTC m=+1933.912841205" watchObservedRunningTime="2026-10-31 00:31:37.277878232 +0000 UTC m=+1933.914696948" Oct 31 00:31:39.208804 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:39.208774 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wmwbh" Oct 31 00:31:50.898816 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:31:50.898778 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:31:52.230511 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:52.230475 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-66476ccc6-t4v98" Oct 31 00:31:52.350881 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:52.350855 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2" Oct 31 00:31:58.264742 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:31:58.264699 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="5e6f500c-b181-456e-b957-4e5defa058f8" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:32:08.263238 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:08.263202 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="5e6f500c-b181-456e-b957-4e5defa058f8" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:32:18.263379 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:18.263349 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:32:20.898877 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:32:20.898830 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:32:50.168272 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.168236 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:32:50.168766 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.168422 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="ad8f04e7-bb53-4453-9967-8bd1df042a74" containerName="otc-container" containerID="cri-o://adaa041afb52b1848236e86b521aa6841aa3a67114e3d36189a2102d8b732c13" gracePeriod=30 Oct 31 00:32:50.409533 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.409510 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:32:50.499074 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.499007 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad8f04e7-bb53-4453-9967-8bd1df042a74-otc-internal\" (UniqueName: \"kubernetes.io/configmap/ad8f04e7-bb53-4453-9967-8bd1df042a74-otc-internal\") pod \"ad8f04e7-bb53-4453-9967-8bd1df042a74\" (UID: \"ad8f04e7-bb53-4453-9967-8bd1df042a74\") " Oct 31 00:32:50.499074 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.499066 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad8f04e7-bb53-4453-9967-8bd1df042a74-kube-api-access-vsxhw\" (UniqueName: \"kubernetes.io/projected/ad8f04e7-bb53-4453-9967-8bd1df042a74-kube-api-access-vsxhw\") pod \"ad8f04e7-bb53-4453-9967-8bd1df042a74\" (UID: \"ad8f04e7-bb53-4453-9967-8bd1df042a74\") " Oct 31 00:32:50.499277 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.499090 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad8f04e7-bb53-4453-9967-8bd1df042a74-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad8f04e7-bb53-4453-9967-8bd1df042a74-otlp-tls-certs\") pod \"ad8f04e7-bb53-4453-9967-8bd1df042a74\" (UID: \"ad8f04e7-bb53-4453-9967-8bd1df042a74\") " Oct 31 00:32:50.499277 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.499125 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad8f04e7-bb53-4453-9967-8bd1df042a74-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad8f04e7-bb53-4453-9967-8bd1df042a74-monitoring-tls-certs\") pod \"ad8f04e7-bb53-4453-9967-8bd1df042a74\" (UID: \"ad8f04e7-bb53-4453-9967-8bd1df042a74\") " Oct 31 00:32:50.499397 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.499322 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad8f04e7-bb53-4453-9967-8bd1df042a74-otc-internal" pod "ad8f04e7-bb53-4453-9967-8bd1df042a74" (UID: "ad8f04e7-bb53-4453-9967-8bd1df042a74"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:50.501173 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.501138 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad8f04e7-bb53-4453-9967-8bd1df042a74-kube-api-access-vsxhw" pod "ad8f04e7-bb53-4453-9967-8bd1df042a74" (UID: "ad8f04e7-bb53-4453-9967-8bd1df042a74"). InnerVolumeSpecName "kube-api-access-vsxhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:50.501173 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.501162 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad8f04e7-bb53-4453-9967-8bd1df042a74-monitoring-tls-certs" pod "ad8f04e7-bb53-4453-9967-8bd1df042a74" (UID: "ad8f04e7-bb53-4453-9967-8bd1df042a74"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:32:50.501356 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.501216 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad8f04e7-bb53-4453-9967-8bd1df042a74-otlp-tls-certs" pod "ad8f04e7-bb53-4453-9967-8bd1df042a74" (UID: "ad8f04e7-bb53-4453-9967-8bd1df042a74"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:32:50.600241 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.600214 5274 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad8f04e7-bb53-4453-9967-8bd1df042a74-monitoring-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:50.600241 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.600242 5274 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ad8f04e7-bb53-4453-9967-8bd1df042a74-otc-internal\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:50.600371 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.600254 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsxhw\" (UniqueName: \"kubernetes.io/projected/ad8f04e7-bb53-4453-9967-8bd1df042a74-kube-api-access-vsxhw\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:50.600371 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.600264 5274 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad8f04e7-bb53-4453-9967-8bd1df042a74-otlp-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:50.618826 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.618803 5274 generic.go:356] "Generic (PLEG): container finished" podID="ad8f04e7-bb53-4453-9967-8bd1df042a74" containerID="adaa041afb52b1848236e86b521aa6841aa3a67114e3d36189a2102d8b732c13" exitCode=0 Oct 31 00:32:50.618929 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.618847 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:32:50.618929 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.618897 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ad8f04e7-bb53-4453-9967-8bd1df042a74","Type":"ContainerDied","Data":"adaa041afb52b1848236e86b521aa6841aa3a67114e3d36189a2102d8b732c13"} Oct 31 00:32:50.619046 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.618946 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ad8f04e7-bb53-4453-9967-8bd1df042a74","Type":"ContainerDied","Data":"9623f202ad296301784e25da11b7f1bda80dbc12bc18dd127b4003278d07a27f"} Oct 31 00:32:50.619046 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.618999 5274 scope.go:122] "RemoveContainer" containerID="adaa041afb52b1848236e86b521aa6841aa3a67114e3d36189a2102d8b732c13" Oct 31 00:32:50.628219 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.628197 5274 scope.go:122] "RemoveContainer" containerID="adaa041afb52b1848236e86b521aa6841aa3a67114e3d36189a2102d8b732c13" Oct 31 00:32:50.628466 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:32:50.628448 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adaa041afb52b1848236e86b521aa6841aa3a67114e3d36189a2102d8b732c13\": container with ID starting with adaa041afb52b1848236e86b521aa6841aa3a67114e3d36189a2102d8b732c13 not found: ID does not exist" containerID="adaa041afb52b1848236e86b521aa6841aa3a67114e3d36189a2102d8b732c13" Oct 31 00:32:50.628519 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.628474 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adaa041afb52b1848236e86b521aa6841aa3a67114e3d36189a2102d8b732c13"} err="failed to get container status \"adaa041afb52b1848236e86b521aa6841aa3a67114e3d36189a2102d8b732c13\": rpc error: code = NotFound desc = could not find container \"adaa041afb52b1848236e86b521aa6841aa3a67114e3d36189a2102d8b732c13\": container with ID starting with adaa041afb52b1848236e86b521aa6841aa3a67114e3d36189a2102d8b732c13 not found: ID does not exist" Oct 31 00:32:50.641190 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.641169 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:32:50.645740 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.645717 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:32:50.675231 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.675208 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:32:50.675568 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.675555 5274 cpu_manager.go:414] "containerMap: removing container" podUID="ad8f04e7-bb53-4453-9967-8bd1df042a74" containerName="otc-container" Oct 31 00:32:50.675613 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.675569 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad8f04e7-bb53-4453-9967-8bd1df042a74" containerName="otc-container" Oct 31 00:32:50.675647 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.675633 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad8f04e7-bb53-4453-9967-8bd1df042a74" containerName="otc-container" Oct 31 00:32:50.679830 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.679815 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:32:50.683527 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.683509 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-rkjdd\"" Oct 31 00:32:50.687055 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.687033 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-54f83bab\"" Oct 31 00:32:50.688989 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.688970 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:32:50.700932 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.700913 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4402f8f1-115f-4439-adb9-985cea9ec05e-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4402f8f1-115f-4439-adb9-985cea9ec05e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:32:50.701047 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.700964 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4402f8f1-115f-4439-adb9-985cea9ec05e-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"4402f8f1-115f-4439-adb9-985cea9ec05e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:32:50.701112 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.701087 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4402f8f1-115f-4439-adb9-985cea9ec05e-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4402f8f1-115f-4439-adb9-985cea9ec05e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:32:50.701212 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.701179 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd5zk\" (UniqueName: \"kubernetes.io/projected/4402f8f1-115f-4439-adb9-985cea9ec05e-kube-api-access-pd5zk\") pod \"data-science-collector-collector-1\" (UID: \"4402f8f1-115f-4439-adb9-985cea9ec05e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:32:50.784367 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.784301 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:32:50.784367 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.784361 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:32:50.801513 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.801487 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4402f8f1-115f-4439-adb9-985cea9ec05e-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"4402f8f1-115f-4439-adb9-985cea9ec05e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:32:50.801617 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.801531 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4402f8f1-115f-4439-adb9-985cea9ec05e-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4402f8f1-115f-4439-adb9-985cea9ec05e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:32:50.801617 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.801576 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd5zk\" (UniqueName: \"kubernetes.io/projected/4402f8f1-115f-4439-adb9-985cea9ec05e-kube-api-access-pd5zk\") pod \"data-science-collector-collector-1\" (UID: \"4402f8f1-115f-4439-adb9-985cea9ec05e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:32:50.801617 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.801597 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4402f8f1-115f-4439-adb9-985cea9ec05e-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4402f8f1-115f-4439-adb9-985cea9ec05e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:32:50.802157 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.802134 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4402f8f1-115f-4439-adb9-985cea9ec05e-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"4402f8f1-115f-4439-adb9-985cea9ec05e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:32:50.804053 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.804037 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4402f8f1-115f-4439-adb9-985cea9ec05e-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4402f8f1-115f-4439-adb9-985cea9ec05e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:32:50.804125 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.804060 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4402f8f1-115f-4439-adb9-985cea9ec05e-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4402f8f1-115f-4439-adb9-985cea9ec05e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:32:50.813727 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.813707 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd5zk\" (UniqueName: \"kubernetes.io/projected/4402f8f1-115f-4439-adb9-985cea9ec05e-kube-api-access-pd5zk\") pod \"data-science-collector-collector-1\" (UID: \"4402f8f1-115f-4439-adb9-985cea9ec05e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:32:50.899135 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:32:50.899110 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:32:50.993540 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:50.993517 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:32:51.266312 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:51.266290 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:32:51.268369 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:32:51.268341 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4402f8f1_115f_4439_adb9_985cea9ec05e.slice/crio-7e1164db53cf246ba3ca5baa4bc854e1626a0728c9c3ed56caa486b221e9c28e WatchSource:0}: Error finding container 7e1164db53cf246ba3ca5baa4bc854e1626a0728c9c3ed56caa486b221e9c28e: Status 404 returned error can't find the container with id 7e1164db53cf246ba3ca5baa4bc854e1626a0728c9c3ed56caa486b221e9c28e Oct 31 00:32:51.270504 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:51.270467 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:32:51.270593 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:51.270526 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:32:51.469622 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:51.469587 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-wmwbh"] Oct 31 00:32:51.469810 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:51.469780 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wmwbh" podUID="591737eb-62c5-45cd-8e05-537ea893a810" containerName="korrel8r" containerID="cri-o://8ad77c08c09cfae07f190503b14f868f44cb0437bd6539f1bcc5c0eb9b3b9ab9" gracePeriod=30 Oct 31 00:32:51.483746 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:51.483721 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad8f04e7-bb53-4453-9967-8bd1df042a74" path="/var/lib/kubelet/pods/ad8f04e7-bb53-4453-9967-8bd1df042a74/volumes" Oct 31 00:32:51.591255 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:51.591218 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:32:51.630905 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:51.630872 5274 generic.go:356] "Generic (PLEG): container finished" podID="591737eb-62c5-45cd-8e05-537ea893a810" containerID="8ad77c08c09cfae07f190503b14f868f44cb0437bd6539f1bcc5c0eb9b3b9ab9" exitCode=0 Oct 31 00:32:51.631095 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:51.631073 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wmwbh" event={"ID":"591737eb-62c5-45cd-8e05-537ea893a810","Type":"ContainerDied","Data":"8ad77c08c09cfae07f190503b14f868f44cb0437bd6539f1bcc5c0eb9b3b9ab9"} Oct 31 00:32:51.634634 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:51.634606 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"4402f8f1-115f-4439-adb9-985cea9ec05e","Type":"ContainerStarted","Data":"9e280a412081508753470c10c906b6d912d3391ccd4491a584ad21bd830da74c"} Oct 31 00:32:51.634778 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:51.634649 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"4402f8f1-115f-4439-adb9-985cea9ec05e","Type":"ContainerStarted","Data":"7e1164db53cf246ba3ca5baa4bc854e1626a0728c9c3ed56caa486b221e9c28e"} Oct 31 00:32:51.634840 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:51.634813 5274 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="redhat-ods-monitoring/data-science-collector-collector-1" secret="" err="secret \"data-science-collector-collector-dockercfg-rkjdd\" not found" Oct 31 00:32:51.656570 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:51.656515 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.656499087 podStartE2EDuration="1.656499087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:32:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:51.654029537 +0000 UTC m=+2008.290848254" watchObservedRunningTime="2026-10-31 00:32:51.656499087 +0000 UTC m=+2008.293317804" Oct 31 00:32:51.711651 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:32:51.711623 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found Oct 31 00:32:51.712390 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:32:51.712352 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4402f8f1-115f-4439-adb9-985cea9ec05e-otlp-tls-certs podName:4402f8f1-115f-4439-adb9-985cea9ec05e nodeName:}" failed. No retries permitted until 2026-10-31 00:32:52.212323827 +0000 UTC m=+2008.849142527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/4402f8f1-115f-4439-adb9-985cea9ec05e-otlp-tls-certs") pod "data-science-collector-collector-1" (UID: "4402f8f1-115f-4439-adb9-985cea9ec05e") : secret "data-science-collector-collector-headless-tls" not found Oct 31 00:32:51.712546 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:32:51.712528 5274 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-collector-collector-54f83bab: configmap "data-science-collector-collector-54f83bab" not found Oct 31 00:32:51.712634 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:32:51.712585 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4402f8f1-115f-4439-adb9-985cea9ec05e-otc-internal podName:4402f8f1-115f-4439-adb9-985cea9ec05e nodeName:}" failed. No retries permitted until 2026-10-31 00:32:52.212570777 +0000 UTC m=+2008.849389470 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otc-internal" (UniqueName: "kubernetes.io/configmap/4402f8f1-115f-4439-adb9-985cea9ec05e-otc-internal") pod "data-science-collector-collector-1" (UID: "4402f8f1-115f-4439-adb9-985cea9ec05e") : configmap "data-science-collector-collector-54f83bab" not found Oct 31 00:32:51.746868 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:51.746844 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wmwbh" Oct 31 00:32:51.801384 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:51.801351 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 31 00:32:51.801621 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:51.801596 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="5e6f500c-b181-456e-b957-4e5defa058f8" containerName="tempo" containerID="cri-o://d3e0b60f787ee6ad1dbfbde5cd69e467810c6191b51554433cbbc9d757c8fb94" gracePeriod=30 Oct 31 00:32:51.813809 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:51.813784 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/591737eb-62c5-45cd-8e05-537ea893a810-kube-api-access-4f5rz\" (UniqueName: \"kubernetes.io/projected/591737eb-62c5-45cd-8e05-537ea893a810-kube-api-access-4f5rz\") pod \"591737eb-62c5-45cd-8e05-537ea893a810\" (UID: \"591737eb-62c5-45cd-8e05-537ea893a810\") " Oct 31 00:32:51.813946 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:51.813921 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/591737eb-62c5-45cd-8e05-537ea893a810-config\" (UniqueName: \"kubernetes.io/configmap/591737eb-62c5-45cd-8e05-537ea893a810-config\") pod \"591737eb-62c5-45cd-8e05-537ea893a810\" (UID: \"591737eb-62c5-45cd-8e05-537ea893a810\") " Oct 31 00:32:51.814035 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:51.814015 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/591737eb-62c5-45cd-8e05-537ea893a810-tls\" (UniqueName: \"kubernetes.io/secret/591737eb-62c5-45cd-8e05-537ea893a810-tls\") pod \"591737eb-62c5-45cd-8e05-537ea893a810\" (UID: \"591737eb-62c5-45cd-8e05-537ea893a810\") " Oct 31 00:32:51.814276 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:51.814252 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/591737eb-62c5-45cd-8e05-537ea893a810-config" pod "591737eb-62c5-45cd-8e05-537ea893a810" (UID: "591737eb-62c5-45cd-8e05-537ea893a810"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:51.816207 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:51.816177 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/591737eb-62c5-45cd-8e05-537ea893a810-tls" pod "591737eb-62c5-45cd-8e05-537ea893a810" (UID: "591737eb-62c5-45cd-8e05-537ea893a810"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:32:51.816306 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:51.816251 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/591737eb-62c5-45cd-8e05-537ea893a810-kube-api-access-4f5rz" pod "591737eb-62c5-45cd-8e05-537ea893a810" (UID: "591737eb-62c5-45cd-8e05-537ea893a810"). InnerVolumeSpecName "kube-api-access-4f5rz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:51.843303 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:51.843231 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-66476ccc6-t4v98"] Oct 31 00:32:51.843601 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:51.843570 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-66476ccc6-t4v98" podUID="d5dece37-633f-4226-a0f0-f672aa15b1e6" containerName="tempo" containerID="cri-o://928b9e1dee765cf1b987c36d65cd2cad9d6714d73bd5441d310d10788ebdf243" gracePeriod=30 Oct 31 00:32:51.848115 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:51.848093 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2"] Oct 31 00:32:51.848281 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:51.848261 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2" podUID="08a395e2-f3b1-405f-bdc2-a324d281136f" containerName="tempo" containerID="cri-o://362acdb4b1f0be1f0c6a672d608b5c21f4fc1fbf2c83bd7d84caa593aa4b0ccf" gracePeriod=30 Oct 31 00:32:51.914668 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:51.914639 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/591737eb-62c5-45cd-8e05-537ea893a810-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:51.914668 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:51.914663 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/591737eb-62c5-45cd-8e05-537ea893a810-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:51.914668 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:51.914673 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f5rz\" (UniqueName: \"kubernetes.io/projected/591737eb-62c5-45cd-8e05-537ea893a810-kube-api-access-4f5rz\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:52.217285 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:32:52.217258 5274 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-collector-collector-54f83bab: configmap "data-science-collector-collector-54f83bab" not found Oct 31 00:32:52.217406 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:32:52.217286 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found Oct 31 00:32:52.217406 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:32:52.217322 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4402f8f1-115f-4439-adb9-985cea9ec05e-otc-internal podName:4402f8f1-115f-4439-adb9-985cea9ec05e nodeName:}" failed. No retries permitted until 2026-10-31 00:32:53.217309604 +0000 UTC m=+2009.854128299 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "otc-internal" (UniqueName: "kubernetes.io/configmap/4402f8f1-115f-4439-adb9-985cea9ec05e-otc-internal") pod "data-science-collector-collector-1" (UID: "4402f8f1-115f-4439-adb9-985cea9ec05e") : configmap "data-science-collector-collector-54f83bab" not found Oct 31 00:32:52.217406 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:32:52.217343 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4402f8f1-115f-4439-adb9-985cea9ec05e-otlp-tls-certs podName:4402f8f1-115f-4439-adb9-985cea9ec05e nodeName:}" failed. No retries permitted until 2026-10-31 00:32:53.217328872 +0000 UTC m=+2009.854147569 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/4402f8f1-115f-4439-adb9-985cea9ec05e-otlp-tls-certs") pod "data-science-collector-collector-1" (UID: "4402f8f1-115f-4439-adb9-985cea9ec05e") : secret "data-science-collector-collector-headless-tls" not found Oct 31 00:32:52.229199 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:52.229163 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-66476ccc6-t4v98" podUID="d5dece37-633f-4226-a0f0-f672aa15b1e6" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:32:52.252220 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:52.252195 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2" podUID="08a395e2-f3b1-405f-bdc2-a324d281136f" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:32:52.641608 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:52.641578 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wmwbh" Oct 31 00:32:52.642036 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:52.641610 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-wmwbh" event={"ID":"591737eb-62c5-45cd-8e05-537ea893a810","Type":"ContainerDied","Data":"c65d84f462a19203f19b6b4db5469907d71b0a617893b6a9c4173a2d1db1462c"} Oct 31 00:32:52.642036 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:52.641666 5274 scope.go:122] "RemoveContainer" containerID="8ad77c08c09cfae07f190503b14f868f44cb0437bd6539f1bcc5c0eb9b3b9ab9" Oct 31 00:32:52.642036 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:52.641810 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="4402f8f1-115f-4439-adb9-985cea9ec05e" containerName="otc-container" containerID="cri-o://9e280a412081508753470c10c906b6d912d3391ccd4491a584ad21bd830da74c" gracePeriod=30 Oct 31 00:32:52.680970 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:52.680938 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-wmwbh"] Oct 31 00:32:52.685523 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:52.685503 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-wmwbh"] Oct 31 00:32:52.902173 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:52.902143 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:32:52.905244 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:52.905225 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-66476ccc6-t4v98" Oct 31 00:32:52.924288 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:52.924265 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4402f8f1-115f-4439-adb9-985cea9ec05e-kube-api-access-pd5zk\" (UniqueName: \"kubernetes.io/projected/4402f8f1-115f-4439-adb9-985cea9ec05e-kube-api-access-pd5zk\") pod \"4402f8f1-115f-4439-adb9-985cea9ec05e\" (UID: \"4402f8f1-115f-4439-adb9-985cea9ec05e\") " Oct 31 00:32:52.924405 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:52.924310 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5dece37-633f-4226-a0f0-f672aa15b1e6-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d5dece37-633f-4226-a0f0-f672aa15b1e6-tempo-conf\") pod \"d5dece37-633f-4226-a0f0-f672aa15b1e6\" (UID: \"d5dece37-633f-4226-a0f0-f672aa15b1e6\") " Oct 31 00:32:52.924405 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:52.924350 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4402f8f1-115f-4439-adb9-985cea9ec05e-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4402f8f1-115f-4439-adb9-985cea9ec05e-otlp-tls-certs\") pod \"4402f8f1-115f-4439-adb9-985cea9ec05e\" (UID: \"4402f8f1-115f-4439-adb9-985cea9ec05e\") " Oct 31 00:32:52.924513 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:52.924408 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4402f8f1-115f-4439-adb9-985cea9ec05e-otc-internal\" (UniqueName: \"kubernetes.io/configmap/4402f8f1-115f-4439-adb9-985cea9ec05e-otc-internal\") pod \"4402f8f1-115f-4439-adb9-985cea9ec05e\" (UID: \"4402f8f1-115f-4439-adb9-985cea9ec05e\") " Oct 31 00:32:52.924513 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:52.924454 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5dece37-633f-4226-a0f0-f672aa15b1e6-tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/d5dece37-633f-4226-a0f0-f672aa15b1e6-tempo-data-science-tempostack-distributor-mtls\") pod \"d5dece37-633f-4226-a0f0-f672aa15b1e6\" (UID: \"d5dece37-633f-4226-a0f0-f672aa15b1e6\") " Oct 31 00:32:52.924513 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:52.924483 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4402f8f1-115f-4439-adb9-985cea9ec05e-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4402f8f1-115f-4439-adb9-985cea9ec05e-monitoring-tls-certs\") pod \"4402f8f1-115f-4439-adb9-985cea9ec05e\" (UID: \"4402f8f1-115f-4439-adb9-985cea9ec05e\") " Oct 31 00:32:52.924655 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:52.924517 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5dece37-633f-4226-a0f0-f672aa15b1e6-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/d5dece37-633f-4226-a0f0-f672aa15b1e6-tempo-tmp-storage\") pod \"d5dece37-633f-4226-a0f0-f672aa15b1e6\" (UID: \"d5dece37-633f-4226-a0f0-f672aa15b1e6\") " Oct 31 00:32:52.924655 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:52.924538 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5dece37-633f-4226-a0f0-f672aa15b1e6-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5dece37-633f-4226-a0f0-f672aa15b1e6-tempo-data-science-tempostack-ca-bundle\") pod \"d5dece37-633f-4226-a0f0-f672aa15b1e6\" (UID: \"d5dece37-633f-4226-a0f0-f672aa15b1e6\") " Oct 31 00:32:52.924655 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:52.924575 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5dece37-633f-4226-a0f0-f672aa15b1e6-kube-api-access-zrc77\" (UniqueName: \"kubernetes.io/projected/d5dece37-633f-4226-a0f0-f672aa15b1e6-kube-api-access-zrc77\") pod \"d5dece37-633f-4226-a0f0-f672aa15b1e6\" (UID: \"d5dece37-633f-4226-a0f0-f672aa15b1e6\") " Oct 31 00:32:52.924890 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:52.924865 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5dece37-633f-4226-a0f0-f672aa15b1e6-tempo-tmp-storage" pod "d5dece37-633f-4226-a0f0-f672aa15b1e6" (UID: "d5dece37-633f-4226-a0f0-f672aa15b1e6"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:52.925133 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:52.925092 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4402f8f1-115f-4439-adb9-985cea9ec05e-otc-internal" pod "4402f8f1-115f-4439-adb9-985cea9ec05e" (UID: "4402f8f1-115f-4439-adb9-985cea9ec05e"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:52.925565 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:52.925534 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5dece37-633f-4226-a0f0-f672aa15b1e6-tempo-data-science-tempostack-ca-bundle" pod "d5dece37-633f-4226-a0f0-f672aa15b1e6" (UID: "d5dece37-633f-4226-a0f0-f672aa15b1e6"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:52.925928 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:52.925882 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5dece37-633f-4226-a0f0-f672aa15b1e6-tempo-conf" pod "d5dece37-633f-4226-a0f0-f672aa15b1e6" (UID: "d5dece37-633f-4226-a0f0-f672aa15b1e6"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:52.926889 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:52.926863 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4402f8f1-115f-4439-adb9-985cea9ec05e-kube-api-access-pd5zk" pod "4402f8f1-115f-4439-adb9-985cea9ec05e" (UID: "4402f8f1-115f-4439-adb9-985cea9ec05e"). InnerVolumeSpecName "kube-api-access-pd5zk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:52.927143 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:52.927100 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5dece37-633f-4226-a0f0-f672aa15b1e6-tempo-data-science-tempostack-distributor-mtls" pod "d5dece37-633f-4226-a0f0-f672aa15b1e6" (UID: "d5dece37-633f-4226-a0f0-f672aa15b1e6"). InnerVolumeSpecName "tempo-data-science-tempostack-distributor-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:32:52.927143 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:52.927114 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4402f8f1-115f-4439-adb9-985cea9ec05e-monitoring-tls-certs" pod "4402f8f1-115f-4439-adb9-985cea9ec05e" (UID: "4402f8f1-115f-4439-adb9-985cea9ec05e"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:32:52.927483 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:52.927442 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4402f8f1-115f-4439-adb9-985cea9ec05e-otlp-tls-certs" pod "4402f8f1-115f-4439-adb9-985cea9ec05e" (UID: "4402f8f1-115f-4439-adb9-985cea9ec05e"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:32:52.928476 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:52.928458 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5dece37-633f-4226-a0f0-f672aa15b1e6-kube-api-access-zrc77" pod "d5dece37-633f-4226-a0f0-f672aa15b1e6" (UID: "d5dece37-633f-4226-a0f0-f672aa15b1e6"). InnerVolumeSpecName "kube-api-access-zrc77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:53.025396 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.025367 5274 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4402f8f1-115f-4439-adb9-985cea9ec05e-otlp-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:53.025396 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.025391 5274 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4402f8f1-115f-4439-adb9-985cea9ec05e-otc-internal\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:53.025592 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.025407 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/d5dece37-633f-4226-a0f0-f672aa15b1e6-tempo-data-science-tempostack-distributor-mtls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:53.025592 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.025424 5274 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4402f8f1-115f-4439-adb9-985cea9ec05e-monitoring-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:53.025592 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.025438 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/d5dece37-633f-4226-a0f0-f672aa15b1e6-tempo-tmp-storage\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:53.025592 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.025450 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5dece37-633f-4226-a0f0-f672aa15b1e6-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:53.025592 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.025463 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrc77\" (UniqueName: \"kubernetes.io/projected/d5dece37-633f-4226-a0f0-f672aa15b1e6-kube-api-access-zrc77\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:53.025592 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.025475 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd5zk\" (UniqueName: \"kubernetes.io/projected/4402f8f1-115f-4439-adb9-985cea9ec05e-kube-api-access-pd5zk\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:53.025592 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.025488 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d5dece37-633f-4226-a0f0-f672aa15b1e6-tempo-conf\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:53.481916 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.481885 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="591737eb-62c5-45cd-8e05-537ea893a810" path="/var/lib/kubelet/pods/591737eb-62c5-45cd-8e05-537ea893a810/volumes" Oct 31 00:32:53.552518 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.552492 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:32:53.632397 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.632366 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e6f500c-b181-456e-b957-4e5defa058f8-tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/5e6f500c-b181-456e-b957-4e5defa058f8-tempo-data-science-tempostack-ingester-mtls\") pod \"5e6f500c-b181-456e-b957-4e5defa058f8\" (UID: \"5e6f500c-b181-456e-b957-4e5defa058f8\") " Oct 31 00:32:53.632521 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.632439 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e6f500c-b181-456e-b957-4e5defa058f8-kube-api-access-9447t\" (UniqueName: \"kubernetes.io/projected/5e6f500c-b181-456e-b957-4e5defa058f8-kube-api-access-9447t\") pod \"5e6f500c-b181-456e-b957-4e5defa058f8\" (UID: \"5e6f500c-b181-456e-b957-4e5defa058f8\") " Oct 31 00:32:53.632599 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.632549 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aac1093750eb10b6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aac1093750eb10b6\") pod \"5e6f500c-b181-456e-b957-4e5defa058f8\" (UID: \"5e6f500c-b181-456e-b957-4e5defa058f8\") " Oct 31 00:32:53.632599 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.632583 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e6f500c-b181-456e-b957-4e5defa058f8-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/5e6f500c-b181-456e-b957-4e5defa058f8-tempo-conf\") pod \"5e6f500c-b181-456e-b957-4e5defa058f8\" (UID: \"5e6f500c-b181-456e-b957-4e5defa058f8\") " Oct 31 00:32:53.632702 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.632662 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e6f500c-b181-456e-b957-4e5defa058f8-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e6f500c-b181-456e-b957-4e5defa058f8-tempo-data-science-tempostack-ca-bundle\") pod \"5e6f500c-b181-456e-b957-4e5defa058f8\" (UID: \"5e6f500c-b181-456e-b957-4e5defa058f8\") " Oct 31 00:32:53.633163 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.633122 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e6f500c-b181-456e-b957-4e5defa058f8-tempo-conf" pod "5e6f500c-b181-456e-b957-4e5defa058f8" (UID: "5e6f500c-b181-456e-b957-4e5defa058f8"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:53.633163 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.633130 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e6f500c-b181-456e-b957-4e5defa058f8-tempo-data-science-tempostack-ca-bundle" pod "5e6f500c-b181-456e-b957-4e5defa058f8" (UID: "5e6f500c-b181-456e-b957-4e5defa058f8"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:53.634563 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.634537 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e6f500c-b181-456e-b957-4e5defa058f8-kube-api-access-9447t" pod "5e6f500c-b181-456e-b957-4e5defa058f8" (UID: "5e6f500c-b181-456e-b957-4e5defa058f8"). InnerVolumeSpecName "kube-api-access-9447t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:53.634563 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.634552 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e6f500c-b181-456e-b957-4e5defa058f8-tempo-data-science-tempostack-ingester-mtls" pod "5e6f500c-b181-456e-b957-4e5defa058f8" (UID: "5e6f500c-b181-456e-b957-4e5defa058f8"). InnerVolumeSpecName "tempo-data-science-tempostack-ingester-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:32:53.635147 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.635127 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0aac1093750eb10b6" pod "5e6f500c-b181-456e-b957-4e5defa058f8" (UID: "5e6f500c-b181-456e-b957-4e5defa058f8"). InnerVolumeSpecName "pvc-1a0c5ca2-2766-4db0-b17f-4a8f45925f49". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:32:53.647612 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.647587 5274 generic.go:356] "Generic (PLEG): container finished" podID="5e6f500c-b181-456e-b957-4e5defa058f8" containerID="d3e0b60f787ee6ad1dbfbde5cd69e467810c6191b51554433cbbc9d757c8fb94" exitCode=0 Oct 31 00:32:53.647920 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.647625 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:32:53.647920 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.647669 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"5e6f500c-b181-456e-b957-4e5defa058f8","Type":"ContainerDied","Data":"d3e0b60f787ee6ad1dbfbde5cd69e467810c6191b51554433cbbc9d757c8fb94"} Oct 31 00:32:53.647920 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.647734 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"5e6f500c-b181-456e-b957-4e5defa058f8","Type":"ContainerDied","Data":"a8e9fa461e879c4dad683f10b466d82e79f40ef656c570b12d19600fe4998e74"} Oct 31 00:32:53.647920 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.647757 5274 scope.go:122] "RemoveContainer" containerID="d3e0b60f787ee6ad1dbfbde5cd69e467810c6191b51554433cbbc9d757c8fb94" Oct 31 00:32:53.650035 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.650013 5274 generic.go:356] "Generic (PLEG): container finished" podID="d5dece37-633f-4226-a0f0-f672aa15b1e6" containerID="928b9e1dee765cf1b987c36d65cd2cad9d6714d73bd5441d310d10788ebdf243" exitCode=0 Oct 31 00:32:53.650130 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.650067 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-66476ccc6-t4v98" Oct 31 00:32:53.650130 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.650082 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-66476ccc6-t4v98" event={"ID":"d5dece37-633f-4226-a0f0-f672aa15b1e6","Type":"ContainerDied","Data":"928b9e1dee765cf1b987c36d65cd2cad9d6714d73bd5441d310d10788ebdf243"} Oct 31 00:32:53.650211 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.650136 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-66476ccc6-t4v98" event={"ID":"d5dece37-633f-4226-a0f0-f672aa15b1e6","Type":"ContainerDied","Data":"0db296a002751532dab5458752e05049b9ef89f9408fb659420da244084d023b"} Oct 31 00:32:53.652216 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.651640 5274 generic.go:356] "Generic (PLEG): container finished" podID="4402f8f1-115f-4439-adb9-985cea9ec05e" containerID="9e280a412081508753470c10c906b6d912d3391ccd4491a584ad21bd830da74c" exitCode=0 Oct 31 00:32:53.652216 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.651681 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"4402f8f1-115f-4439-adb9-985cea9ec05e","Type":"ContainerDied","Data":"9e280a412081508753470c10c906b6d912d3391ccd4491a584ad21bd830da74c"} Oct 31 00:32:53.652216 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.651709 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:32:53.652216 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.651717 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"4402f8f1-115f-4439-adb9-985cea9ec05e","Type":"ContainerDied","Data":"7e1164db53cf246ba3ca5baa4bc854e1626a0728c9c3ed56caa486b221e9c28e"} Oct 31 00:32:53.657903 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.657887 5274 scope.go:122] "RemoveContainer" containerID="d3e0b60f787ee6ad1dbfbde5cd69e467810c6191b51554433cbbc9d757c8fb94" Oct 31 00:32:53.658191 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:32:53.658170 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3e0b60f787ee6ad1dbfbde5cd69e467810c6191b51554433cbbc9d757c8fb94\": container with ID starting with d3e0b60f787ee6ad1dbfbde5cd69e467810c6191b51554433cbbc9d757c8fb94 not found: ID does not exist" containerID="d3e0b60f787ee6ad1dbfbde5cd69e467810c6191b51554433cbbc9d757c8fb94" Oct 31 00:32:53.658259 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.658201 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3e0b60f787ee6ad1dbfbde5cd69e467810c6191b51554433cbbc9d757c8fb94"} err="failed to get container status \"d3e0b60f787ee6ad1dbfbde5cd69e467810c6191b51554433cbbc9d757c8fb94\": rpc error: code = NotFound desc = could not find container \"d3e0b60f787ee6ad1dbfbde5cd69e467810c6191b51554433cbbc9d757c8fb94\": container with ID starting with d3e0b60f787ee6ad1dbfbde5cd69e467810c6191b51554433cbbc9d757c8fb94 not found: ID does not exist" Oct 31 00:32:53.658259 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.658222 5274 scope.go:122] "RemoveContainer" containerID="928b9e1dee765cf1b987c36d65cd2cad9d6714d73bd5441d310d10788ebdf243" Oct 31 00:32:53.665580 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.665565 5274 scope.go:122] "RemoveContainer" containerID="928b9e1dee765cf1b987c36d65cd2cad9d6714d73bd5441d310d10788ebdf243" Oct 31 00:32:53.665818 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:32:53.665802 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"928b9e1dee765cf1b987c36d65cd2cad9d6714d73bd5441d310d10788ebdf243\": container with ID starting with 928b9e1dee765cf1b987c36d65cd2cad9d6714d73bd5441d310d10788ebdf243 not found: ID does not exist" containerID="928b9e1dee765cf1b987c36d65cd2cad9d6714d73bd5441d310d10788ebdf243" Oct 31 00:32:53.665863 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.665824 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"928b9e1dee765cf1b987c36d65cd2cad9d6714d73bd5441d310d10788ebdf243"} err="failed to get container status \"928b9e1dee765cf1b987c36d65cd2cad9d6714d73bd5441d310d10788ebdf243\": rpc error: code = NotFound desc = could not find container \"928b9e1dee765cf1b987c36d65cd2cad9d6714d73bd5441d310d10788ebdf243\": container with ID starting with 928b9e1dee765cf1b987c36d65cd2cad9d6714d73bd5441d310d10788ebdf243 not found: ID does not exist" Oct 31 00:32:53.665863 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.665840 5274 scope.go:122] "RemoveContainer" containerID="9e280a412081508753470c10c906b6d912d3391ccd4491a584ad21bd830da74c" Oct 31 00:32:53.673395 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.673374 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:32:53.673709 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.673697 5274 scope.go:122] "RemoveContainer" containerID="9e280a412081508753470c10c906b6d912d3391ccd4491a584ad21bd830da74c" Oct 31 00:32:53.673937 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:32:53.673920 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e280a412081508753470c10c906b6d912d3391ccd4491a584ad21bd830da74c\": container with ID starting with 9e280a412081508753470c10c906b6d912d3391ccd4491a584ad21bd830da74c not found: ID does not exist" containerID="9e280a412081508753470c10c906b6d912d3391ccd4491a584ad21bd830da74c" Oct 31 00:32:53.674021 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.673944 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e280a412081508753470c10c906b6d912d3391ccd4491a584ad21bd830da74c"} err="failed to get container status \"9e280a412081508753470c10c906b6d912d3391ccd4491a584ad21bd830da74c\": rpc error: code = NotFound desc = could not find container \"9e280a412081508753470c10c906b6d912d3391ccd4491a584ad21bd830da74c\": container with ID starting with 9e280a412081508753470c10c906b6d912d3391ccd4491a584ad21bd830da74c not found: ID does not exist" Oct 31 00:32:53.677633 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.677615 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:32:53.690480 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.690461 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 31 00:32:53.695236 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.695217 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 31 00:32:53.707368 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.707347 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-66476ccc6-t4v98"] Oct 31 00:32:53.711530 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.711513 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-66476ccc6-t4v98"] Oct 31 00:32:53.733885 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.733842 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/5e6f500c-b181-456e-b957-4e5defa058f8-tempo-data-science-tempostack-ingester-mtls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:53.733885 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.733861 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9447t\" (UniqueName: \"kubernetes.io/projected/5e6f500c-b181-456e-b957-4e5defa058f8-kube-api-access-9447t\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:53.734020 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.733889 5274 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-1a0c5ca2-2766-4db0-b17f-4a8f45925f49\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aac1093750eb10b6\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" " Oct 31 00:32:53.734020 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.733902 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/5e6f500c-b181-456e-b957-4e5defa058f8-tempo-conf\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:53.734020 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.733912 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e6f500c-b181-456e-b957-4e5defa058f8-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:53.742913 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.742901 5274 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-1a0c5ca2-2766-4db0-b17f-4a8f45925f49" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0aac1093750eb10b6") on node "ip-10-0-129-70.us-east-2.compute.internal" Oct 31 00:32:53.835019 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:53.834998 5274 reconciler_common.go:299] "Volume detached for volume \"pvc-1a0c5ca2-2766-4db0-b17f-4a8f45925f49\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aac1093750eb10b6\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"csi-8a41e48b52e68bf1c119110f180532301f2a775d1f1f46a0f1f5e5aa4e023bca\"" Oct 31 00:32:55.480776 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:55.480746 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4402f8f1-115f-4439-adb9-985cea9ec05e" path="/var/lib/kubelet/pods/4402f8f1-115f-4439-adb9-985cea9ec05e/volumes" Oct 31 00:32:55.481186 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:55.481155 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e6f500c-b181-456e-b957-4e5defa058f8" path="/var/lib/kubelet/pods/5e6f500c-b181-456e-b957-4e5defa058f8/volumes" Oct 31 00:32:55.481520 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:32:55.481507 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5dece37-633f-4226-a0f0-f672aa15b1e6" path="/var/lib/kubelet/pods/d5dece37-633f-4226-a0f0-f672aa15b1e6/volumes" Oct 31 00:33:02.249663 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:02.249622 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2" podUID="08a395e2-f3b1-405f-bdc2-a324d281136f" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:33:12.250537 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:12.250494 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2" podUID="08a395e2-f3b1-405f-bdc2-a324d281136f" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:33:12.251085 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:12.250618 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2" Oct 31 00:33:20.898922 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:33:20.898885 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:33:22.014770 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:22.014742 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2" Oct 31 00:33:22.153301 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:22.153271 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08a395e2-f3b1-405f-bdc2-a324d281136f-kube-api-access-7kd48\" (UniqueName: \"kubernetes.io/projected/08a395e2-f3b1-405f-bdc2-a324d281136f-kube-api-access-7kd48\") pod \"08a395e2-f3b1-405f-bdc2-a324d281136f\" (UID: \"08a395e2-f3b1-405f-bdc2-a324d281136f\") " Oct 31 00:33:22.153461 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:22.153305 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08a395e2-f3b1-405f-bdc2-a324d281136f-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/08a395e2-f3b1-405f-bdc2-a324d281136f-tempo-conf\") pod \"08a395e2-f3b1-405f-bdc2-a324d281136f\" (UID: \"08a395e2-f3b1-405f-bdc2-a324d281136f\") " Oct 31 00:33:22.153461 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:22.153329 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08a395e2-f3b1-405f-bdc2-a324d281136f-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/08a395e2-f3b1-405f-bdc2-a324d281136f-tempo-tmp-storage\") pod \"08a395e2-f3b1-405f-bdc2-a324d281136f\" (UID: \"08a395e2-f3b1-405f-bdc2-a324d281136f\") " Oct 31 00:33:22.153461 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:22.153369 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08a395e2-f3b1-405f-bdc2-a324d281136f-tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/08a395e2-f3b1-405f-bdc2-a324d281136f-tempo-data-science-tempostack-query-frontend-mtls\") pod \"08a395e2-f3b1-405f-bdc2-a324d281136f\" (UID: \"08a395e2-f3b1-405f-bdc2-a324d281136f\") " Oct 31 00:33:22.153461 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:22.153404 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08a395e2-f3b1-405f-bdc2-a324d281136f-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08a395e2-f3b1-405f-bdc2-a324d281136f-tempo-data-science-tempostack-ca-bundle\") pod \"08a395e2-f3b1-405f-bdc2-a324d281136f\" (UID: \"08a395e2-f3b1-405f-bdc2-a324d281136f\") " Oct 31 00:33:22.153841 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:22.153811 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08a395e2-f3b1-405f-bdc2-a324d281136f-tempo-conf" pod "08a395e2-f3b1-405f-bdc2-a324d281136f" (UID: "08a395e2-f3b1-405f-bdc2-a324d281136f"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:33:22.153841 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:22.153819 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a395e2-f3b1-405f-bdc2-a324d281136f-tempo-tmp-storage" pod "08a395e2-f3b1-405f-bdc2-a324d281136f" (UID: "08a395e2-f3b1-405f-bdc2-a324d281136f"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:22.154027 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:22.153848 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08a395e2-f3b1-405f-bdc2-a324d281136f-tempo-data-science-tempostack-ca-bundle" pod "08a395e2-f3b1-405f-bdc2-a324d281136f" (UID: "08a395e2-f3b1-405f-bdc2-a324d281136f"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:33:22.155474 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:22.155450 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08a395e2-f3b1-405f-bdc2-a324d281136f-kube-api-access-7kd48" pod "08a395e2-f3b1-405f-bdc2-a324d281136f" (UID: "08a395e2-f3b1-405f-bdc2-a324d281136f"). InnerVolumeSpecName "kube-api-access-7kd48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:33:22.155580 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:22.155538 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08a395e2-f3b1-405f-bdc2-a324d281136f-tempo-data-science-tempostack-query-frontend-mtls" pod "08a395e2-f3b1-405f-bdc2-a324d281136f" (UID: "08a395e2-f3b1-405f-bdc2-a324d281136f"). InnerVolumeSpecName "tempo-data-science-tempostack-query-frontend-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:33:22.253976 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:22.253932 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08a395e2-f3b1-405f-bdc2-a324d281136f-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:22.253976 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:22.253965 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kd48\" (UniqueName: \"kubernetes.io/projected/08a395e2-f3b1-405f-bdc2-a324d281136f-kube-api-access-7kd48\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:22.254139 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:22.253985 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/08a395e2-f3b1-405f-bdc2-a324d281136f-tempo-conf\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:22.254139 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:22.253994 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/08a395e2-f3b1-405f-bdc2-a324d281136f-tempo-tmp-storage\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:22.254139 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:22.254003 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/08a395e2-f3b1-405f-bdc2-a324d281136f-tempo-data-science-tempostack-query-frontend-mtls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:22.800251 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:22.800218 5274 generic.go:356] "Generic (PLEG): container finished" podID="08a395e2-f3b1-405f-bdc2-a324d281136f" containerID="362acdb4b1f0be1f0c6a672d608b5c21f4fc1fbf2c83bd7d84caa593aa4b0ccf" exitCode=137 Oct 31 00:33:22.800425 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:22.800268 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2" Oct 31 00:33:22.800425 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:22.800297 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2" event={"ID":"08a395e2-f3b1-405f-bdc2-a324d281136f","Type":"ContainerDied","Data":"362acdb4b1f0be1f0c6a672d608b5c21f4fc1fbf2c83bd7d84caa593aa4b0ccf"} Oct 31 00:33:22.800425 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:22.800352 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2" event={"ID":"08a395e2-f3b1-405f-bdc2-a324d281136f","Type":"ContainerDied","Data":"994f243ab1bf645aa6b28899cc8e1e30049a70953771ed1e21729eec234b752c"} Oct 31 00:33:22.800425 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:22.800373 5274 scope.go:122] "RemoveContainer" containerID="362acdb4b1f0be1f0c6a672d608b5c21f4fc1fbf2c83bd7d84caa593aa4b0ccf" Oct 31 00:33:22.809223 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:22.809208 5274 scope.go:122] "RemoveContainer" containerID="362acdb4b1f0be1f0c6a672d608b5c21f4fc1fbf2c83bd7d84caa593aa4b0ccf" Oct 31 00:33:22.809496 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:33:22.809479 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"362acdb4b1f0be1f0c6a672d608b5c21f4fc1fbf2c83bd7d84caa593aa4b0ccf\": container with ID starting with 362acdb4b1f0be1f0c6a672d608b5c21f4fc1fbf2c83bd7d84caa593aa4b0ccf not found: ID does not exist" containerID="362acdb4b1f0be1f0c6a672d608b5c21f4fc1fbf2c83bd7d84caa593aa4b0ccf" Oct 31 00:33:22.809570 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:22.809504 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"362acdb4b1f0be1f0c6a672d608b5c21f4fc1fbf2c83bd7d84caa593aa4b0ccf"} err="failed to get container status \"362acdb4b1f0be1f0c6a672d608b5c21f4fc1fbf2c83bd7d84caa593aa4b0ccf\": rpc error: code = NotFound desc = could not find container \"362acdb4b1f0be1f0c6a672d608b5c21f4fc1fbf2c83bd7d84caa593aa4b0ccf\": container with ID starting with 362acdb4b1f0be1f0c6a672d608b5c21f4fc1fbf2c83bd7d84caa593aa4b0ccf not found: ID does not exist" Oct 31 00:33:22.823228 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:22.823205 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2"] Oct 31 00:33:22.827600 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:22.827580 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7559db57dd-s85k2"] Oct 31 00:33:23.481715 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:23.481678 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08a395e2-f3b1-405f-bdc2-a324d281136f" path="/var/lib/kubelet/pods/08a395e2-f3b1-405f-bdc2-a324d281136f/volumes" Oct 31 00:33:35.159609 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.159577 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-q62ss"] Oct 31 00:33:35.160137 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.160026 5274 cpu_manager.go:414] "containerMap: removing container" podUID="5e6f500c-b181-456e-b957-4e5defa058f8" containerName="tempo" Oct 31 00:33:35.160137 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.160059 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e6f500c-b181-456e-b957-4e5defa058f8" containerName="tempo" Oct 31 00:33:35.160137 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.160074 5274 cpu_manager.go:414] "containerMap: removing container" podUID="591737eb-62c5-45cd-8e05-537ea893a810" containerName="korrel8r" Oct 31 00:33:35.160137 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.160081 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="591737eb-62c5-45cd-8e05-537ea893a810" containerName="korrel8r" Oct 31 00:33:35.160137 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.160089 5274 cpu_manager.go:414] "containerMap: removing container" podUID="08a395e2-f3b1-405f-bdc2-a324d281136f" containerName="tempo" Oct 31 00:33:35.160137 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.160094 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08a395e2-f3b1-405f-bdc2-a324d281136f" containerName="tempo" Oct 31 00:33:35.160137 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.160102 5274 cpu_manager.go:414] "containerMap: removing container" podUID="4402f8f1-115f-4439-adb9-985cea9ec05e" containerName="otc-container" Oct 31 00:33:35.160137 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.160107 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4402f8f1-115f-4439-adb9-985cea9ec05e" containerName="otc-container" Oct 31 00:33:35.160137 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.160118 5274 cpu_manager.go:414] "containerMap: removing container" podUID="d5dece37-633f-4226-a0f0-f672aa15b1e6" containerName="tempo" Oct 31 00:33:35.160137 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.160123 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5dece37-633f-4226-a0f0-f672aa15b1e6" containerName="tempo" Oct 31 00:33:35.160445 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.160181 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="4402f8f1-115f-4439-adb9-985cea9ec05e" containerName="otc-container" Oct 31 00:33:35.160445 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.160189 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5dece37-633f-4226-a0f0-f672aa15b1e6" containerName="tempo" Oct 31 00:33:35.160445 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.160198 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e6f500c-b181-456e-b957-4e5defa058f8" containerName="tempo" Oct 31 00:33:35.160445 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.160204 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="591737eb-62c5-45cd-8e05-537ea893a810" containerName="korrel8r" Oct 31 00:33:35.160445 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.160209 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="08a395e2-f3b1-405f-bdc2-a324d281136f" containerName="tempo" Oct 31 00:33:35.164697 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.164681 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-q62ss" Oct 31 00:33:35.172391 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.172363 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-q62ss"] Oct 31 00:33:35.173548 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.173515 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 31 00:33:35.173648 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.173554 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:33:35.173648 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.173612 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:33:35.173648 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.173614 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 31 00:33:35.173757 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.173697 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-4j5v7\"" Oct 31 00:33:35.251386 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.251358 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/27ca4a05-d89b-40c7-a71c-678ffa630bda-config\") pod \"korrel8r-5d45699dc4-q62ss\" (UID: \"27ca4a05-d89b-40c7-a71c-678ffa630bda\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-q62ss" Oct 31 00:33:35.251524 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.251402 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t56f7\" (UniqueName: \"kubernetes.io/projected/27ca4a05-d89b-40c7-a71c-678ffa630bda-kube-api-access-t56f7\") pod \"korrel8r-5d45699dc4-q62ss\" (UID: \"27ca4a05-d89b-40c7-a71c-678ffa630bda\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-q62ss" Oct 31 00:33:35.251524 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.251427 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/27ca4a05-d89b-40c7-a71c-678ffa630bda-tls\") pod \"korrel8r-5d45699dc4-q62ss\" (UID: \"27ca4a05-d89b-40c7-a71c-678ffa630bda\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-q62ss" Oct 31 00:33:35.352590 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.352559 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/27ca4a05-d89b-40c7-a71c-678ffa630bda-config\") pod \"korrel8r-5d45699dc4-q62ss\" (UID: \"27ca4a05-d89b-40c7-a71c-678ffa630bda\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-q62ss" Oct 31 00:33:35.352749 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.352605 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t56f7\" (UniqueName: \"kubernetes.io/projected/27ca4a05-d89b-40c7-a71c-678ffa630bda-kube-api-access-t56f7\") pod \"korrel8r-5d45699dc4-q62ss\" (UID: \"27ca4a05-d89b-40c7-a71c-678ffa630bda\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-q62ss" Oct 31 00:33:35.352749 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.352639 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/27ca4a05-d89b-40c7-a71c-678ffa630bda-tls\") pod \"korrel8r-5d45699dc4-q62ss\" (UID: \"27ca4a05-d89b-40c7-a71c-678ffa630bda\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-q62ss" Oct 31 00:33:35.353405 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.353385 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/27ca4a05-d89b-40c7-a71c-678ffa630bda-config\") pod \"korrel8r-5d45699dc4-q62ss\" (UID: \"27ca4a05-d89b-40c7-a71c-678ffa630bda\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-q62ss" Oct 31 00:33:35.355138 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.355114 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/27ca4a05-d89b-40c7-a71c-678ffa630bda-tls\") pod \"korrel8r-5d45699dc4-q62ss\" (UID: \"27ca4a05-d89b-40c7-a71c-678ffa630bda\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-q62ss" Oct 31 00:33:35.370318 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.370286 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t56f7\" (UniqueName: \"kubernetes.io/projected/27ca4a05-d89b-40c7-a71c-678ffa630bda-kube-api-access-t56f7\") pod \"korrel8r-5d45699dc4-q62ss\" (UID: \"27ca4a05-d89b-40c7-a71c-678ffa630bda\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-q62ss" Oct 31 00:33:35.455078 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.454989 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:33:35.459186 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.459160 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:33:35.473693 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.473664 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:33:35.478200 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.478178 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-q62ss" Oct 31 00:33:35.537433 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.537404 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:33:35.537614 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.537485 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:33:35.537614 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.537522 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-w5bd4\"" Oct 31 00:33:35.554387 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.554357 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb91015f-7854-47ed-9d1b-b81a71661356-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"cb91015f-7854-47ed-9d1b-b81a71661356\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:33:35.554479 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.554411 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb91015f-7854-47ed-9d1b-b81a71661356-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"cb91015f-7854-47ed-9d1b-b81a71661356\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:33:35.554479 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.554445 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cb91015f-7854-47ed-9d1b-b81a71661356-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"cb91015f-7854-47ed-9d1b-b81a71661356\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:33:35.554479 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.554472 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbmlm\" (UniqueName: \"kubernetes.io/projected/cb91015f-7854-47ed-9d1b-b81a71661356-kube-api-access-tbmlm\") pod \"data-science-collector-collector-1\" (UID: \"cb91015f-7854-47ed-9d1b-b81a71661356\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:33:35.557792 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.557766 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-54f83bab\"" Oct 31 00:33:35.655463 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.655431 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb91015f-7854-47ed-9d1b-b81a71661356-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"cb91015f-7854-47ed-9d1b-b81a71661356\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:33:35.655596 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.655472 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb91015f-7854-47ed-9d1b-b81a71661356-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"cb91015f-7854-47ed-9d1b-b81a71661356\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:33:35.655596 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.655496 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cb91015f-7854-47ed-9d1b-b81a71661356-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"cb91015f-7854-47ed-9d1b-b81a71661356\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:33:35.655596 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.655568 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbmlm\" (UniqueName: \"kubernetes.io/projected/cb91015f-7854-47ed-9d1b-b81a71661356-kube-api-access-tbmlm\") pod \"data-science-collector-collector-1\" (UID: \"cb91015f-7854-47ed-9d1b-b81a71661356\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:33:35.656243 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.656219 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cb91015f-7854-47ed-9d1b-b81a71661356-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"cb91015f-7854-47ed-9d1b-b81a71661356\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:33:35.657894 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.657872 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb91015f-7854-47ed-9d1b-b81a71661356-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"cb91015f-7854-47ed-9d1b-b81a71661356\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:33:35.658017 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.657928 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb91015f-7854-47ed-9d1b-b81a71661356-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"cb91015f-7854-47ed-9d1b-b81a71661356\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:33:35.668557 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.668537 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbmlm\" (UniqueName: \"kubernetes.io/projected/cb91015f-7854-47ed-9d1b-b81a71661356-kube-api-access-tbmlm\") pod \"data-science-collector-collector-1\" (UID: \"cb91015f-7854-47ed-9d1b-b81a71661356\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:33:35.771058 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.771032 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-q62ss"] Oct 31 00:33:35.773667 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:33:35.773632 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ca4a05_d89b_40c7_a71c_678ffa630bda.slice/crio-844d4a5b51c4dac740ccb2c650aa31c98be288c8e3ef7f0e492d961279ae9bae WatchSource:0}: Error finding container 844d4a5b51c4dac740ccb2c650aa31c98be288c8e3ef7f0e492d961279ae9bae: Status 404 returned error can't find the container with id 844d4a5b51c4dac740ccb2c650aa31c98be288c8e3ef7f0e492d961279ae9bae Oct 31 00:33:35.776259 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.776242 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:33:35.865878 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.865845 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-q62ss" event={"ID":"27ca4a05-d89b-40c7-a71c-678ffa630bda","Type":"ContainerStarted","Data":"27ea08e42682a00433f120251b124b92d8de58ca3735d8aa99dd5eacb1a77bc5"} Oct 31 00:33:35.866030 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.865902 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-q62ss" event={"ID":"27ca4a05-d89b-40c7-a71c-678ffa630bda","Type":"ContainerStarted","Data":"844d4a5b51c4dac740ccb2c650aa31c98be288c8e3ef7f0e492d961279ae9bae"} Oct 31 00:33:35.866106 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.866047 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-q62ss" Oct 31 00:33:35.885906 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:35.885860 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-q62ss" podStartSLOduration=0.885846929 podStartE2EDuration="885.846929ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:35.88513591 +0000 UTC m=+2052.521954627" watchObservedRunningTime="2026-10-31 00:33:35.885846929 +0000 UTC m=+2052.522665646" Oct 31 00:33:36.061045 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:36.060943 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:33:36.063827 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:33:36.063798 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb91015f_7854_47ed_9d1b_b81a71661356.slice/crio-83d50264cb3a1667ad372b036198925d03f382d8b0a62679d9a373a0a5f52b48 WatchSource:0}: Error finding container 83d50264cb3a1667ad372b036198925d03f382d8b0a62679d9a373a0a5f52b48: Status 404 returned error can't find the container with id 83d50264cb3a1667ad372b036198925d03f382d8b0a62679d9a373a0a5f52b48 Oct 31 00:33:36.066041 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:36.066006 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:33:36.066118 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:36.066074 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:33:36.873889 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:36.873851 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"cb91015f-7854-47ed-9d1b-b81a71661356","Type":"ContainerStarted","Data":"c4079bca86cd83bcfc1c13cbc6ce9727738aa31d43f18d7c03aabbe79b8cacea"} Oct 31 00:33:36.874339 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:36.873914 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"cb91015f-7854-47ed-9d1b-b81a71661356","Type":"ContainerStarted","Data":"83d50264cb3a1667ad372b036198925d03f382d8b0a62679d9a373a0a5f52b48"} Oct 31 00:33:36.901647 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:36.898463 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.898445062 podStartE2EDuration="1.898445062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:36.895798194 +0000 UTC m=+2053.532616909" watchObservedRunningTime="2026-10-31 00:33:36.898445062 +0000 UTC m=+2053.535263779" Oct 31 00:33:38.136189 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.136157 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h"] Oct 31 00:33:38.139775 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.139755 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h" Oct 31 00:33:38.149098 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.148308 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-ca-bundle\"" Oct 31 00:33:38.149098 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.148476 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack\"" Oct 31 00:33:38.149804 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.149778 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp"] Oct 31 00:33:38.155788 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.155769 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h"] Oct 31 00:33:38.155936 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.155864 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp" Oct 31 00:33:38.171036 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.169382 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp"] Oct 31 00:33:38.176381 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.176359 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"gcs-secret\"" Oct 31 00:33:38.176762 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.176741 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-dockercfg-4n979\"" Oct 31 00:33:38.176846 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.176780 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-query-frontend-mtls\"" Oct 31 00:33:38.176846 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.176816 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-querier-mtls\"" Oct 31 00:33:38.186991 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.186751 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8"] Oct 31 00:33:38.190786 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.190765 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8" Oct 31 00:33:38.202380 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.202354 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8"] Oct 31 00:33:38.239504 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.239472 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-compactor-mtls\"" Oct 31 00:33:38.278808 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.278783 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/f26817af-cbdf-4baa-86b0-8b21f395779a-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h\" (UID: \"f26817af-cbdf-4baa-86b0-8b21f395779a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h" Oct 31 00:33:38.278808 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.278813 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68837eb6-0f8f-4215-8771-1d661393310c-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8\" (UID: \"68837eb6-0f8f-4215-8771-1d661393310c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8" Oct 31 00:33:38.279026 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.278839 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/68837eb6-0f8f-4215-8771-1d661393310c-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8\" (UID: \"68837eb6-0f8f-4215-8771-1d661393310c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8" Oct 31 00:33:38.279026 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.278858 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/f26817af-cbdf-4baa-86b0-8b21f395779a-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h\" (UID: \"f26817af-cbdf-4baa-86b0-8b21f395779a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h" Oct 31 00:33:38.279026 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.278897 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/68837eb6-0f8f-4215-8771-1d661393310c-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8\" (UID: \"68837eb6-0f8f-4215-8771-1d661393310c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8" Oct 31 00:33:38.279026 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.278989 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/2676a8db-5e75-400f-b0af-5b617d4204a3-tempo-conf\") pod \"tempo-data-science-tempostack-querier-7f969fccfb-xbjdp\" (UID: \"2676a8db-5e75-400f-b0af-5b617d4204a3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp" Oct 31 00:33:38.279026 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.279011 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgz2l\" (UniqueName: \"kubernetes.io/projected/f26817af-cbdf-4baa-86b0-8b21f395779a-kube-api-access-vgz2l\") pod \"tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h\" (UID: \"f26817af-cbdf-4baa-86b0-8b21f395779a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h" Oct 31 00:33:38.279213 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.279033 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/2676a8db-5e75-400f-b0af-5b617d4204a3-storage-gcs-key\") pod \"tempo-data-science-tempostack-querier-7f969fccfb-xbjdp\" (UID: \"2676a8db-5e75-400f-b0af-5b617d4204a3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp" Oct 31 00:33:38.279213 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.279104 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwmpf\" (UniqueName: \"kubernetes.io/projected/2676a8db-5e75-400f-b0af-5b617d4204a3-kube-api-access-bwmpf\") pod \"tempo-data-science-tempostack-querier-7f969fccfb-xbjdp\" (UID: \"2676a8db-5e75-400f-b0af-5b617d4204a3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp" Oct 31 00:33:38.279213 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.279140 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f26817af-cbdf-4baa-86b0-8b21f395779a-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h\" (UID: \"f26817af-cbdf-4baa-86b0-8b21f395779a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h" Oct 31 00:33:38.279213 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.279180 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/2676a8db-5e75-400f-b0af-5b617d4204a3-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-7f969fccfb-xbjdp\" (UID: \"2676a8db-5e75-400f-b0af-5b617d4204a3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp" Oct 31 00:33:38.279213 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.279206 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2676a8db-5e75-400f-b0af-5b617d4204a3-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-7f969fccfb-xbjdp\" (UID: \"2676a8db-5e75-400f-b0af-5b617d4204a3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp" Oct 31 00:33:38.279385 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.279232 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/f26817af-cbdf-4baa-86b0-8b21f395779a-storage-gcs-key\") pod \"tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h\" (UID: \"f26817af-cbdf-4baa-86b0-8b21f395779a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h" Oct 31 00:33:38.279385 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.279265 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/2676a8db-5e75-400f-b0af-5b617d4204a3-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-7f969fccfb-xbjdp\" (UID: \"2676a8db-5e75-400f-b0af-5b617d4204a3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp" Oct 31 00:33:38.279385 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.279317 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f26817af-cbdf-4baa-86b0-8b21f395779a-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h\" (UID: \"f26817af-cbdf-4baa-86b0-8b21f395779a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h" Oct 31 00:33:38.279385 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.279350 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/68837eb6-0f8f-4215-8771-1d661393310c-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8\" (UID: \"68837eb6-0f8f-4215-8771-1d661393310c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8" Oct 31 00:33:38.279385 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.279366 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/68837eb6-0f8f-4215-8771-1d661393310c-storage-gcs-key\") pod \"tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8\" (UID: \"68837eb6-0f8f-4215-8771-1d661393310c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8" Oct 31 00:33:38.279544 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.279397 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhkdj\" (UniqueName: \"kubernetes.io/projected/68837eb6-0f8f-4215-8771-1d661393310c-kube-api-access-zhkdj\") pod \"tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8\" (UID: \"68837eb6-0f8f-4215-8771-1d661393310c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8" Oct 31 00:33:38.380142 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.380111 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/68837eb6-0f8f-4215-8771-1d661393310c-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8\" (UID: \"68837eb6-0f8f-4215-8771-1d661393310c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8" Oct 31 00:33:38.380286 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.380143 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/68837eb6-0f8f-4215-8771-1d661393310c-storage-gcs-key\") pod \"tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8\" (UID: \"68837eb6-0f8f-4215-8771-1d661393310c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8" Oct 31 00:33:38.380286 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.380172 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhkdj\" (UniqueName: \"kubernetes.io/projected/68837eb6-0f8f-4215-8771-1d661393310c-kube-api-access-zhkdj\") pod \"tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8\" (UID: \"68837eb6-0f8f-4215-8771-1d661393310c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8" Oct 31 00:33:38.380286 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.380195 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/f26817af-cbdf-4baa-86b0-8b21f395779a-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h\" (UID: \"f26817af-cbdf-4baa-86b0-8b21f395779a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h" Oct 31 00:33:38.380286 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.380219 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68837eb6-0f8f-4215-8771-1d661393310c-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8\" (UID: \"68837eb6-0f8f-4215-8771-1d661393310c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8" Oct 31 00:33:38.380286 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.380258 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/68837eb6-0f8f-4215-8771-1d661393310c-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8\" (UID: \"68837eb6-0f8f-4215-8771-1d661393310c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8" Oct 31 00:33:38.380551 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.380285 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/f26817af-cbdf-4baa-86b0-8b21f395779a-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h\" (UID: \"f26817af-cbdf-4baa-86b0-8b21f395779a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h" Oct 31 00:33:38.380551 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.380314 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/68837eb6-0f8f-4215-8771-1d661393310c-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8\" (UID: \"68837eb6-0f8f-4215-8771-1d661393310c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8" Oct 31 00:33:38.380551 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.380350 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/2676a8db-5e75-400f-b0af-5b617d4204a3-tempo-conf\") pod \"tempo-data-science-tempostack-querier-7f969fccfb-xbjdp\" (UID: \"2676a8db-5e75-400f-b0af-5b617d4204a3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp" Oct 31 00:33:38.380551 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.380377 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgz2l\" (UniqueName: \"kubernetes.io/projected/f26817af-cbdf-4baa-86b0-8b21f395779a-kube-api-access-vgz2l\") pod \"tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h\" (UID: \"f26817af-cbdf-4baa-86b0-8b21f395779a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h" Oct 31 00:33:38.380551 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.380413 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/2676a8db-5e75-400f-b0af-5b617d4204a3-storage-gcs-key\") pod \"tempo-data-science-tempostack-querier-7f969fccfb-xbjdp\" (UID: \"2676a8db-5e75-400f-b0af-5b617d4204a3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp" Oct 31 00:33:38.380551 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.380458 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwmpf\" (UniqueName: \"kubernetes.io/projected/2676a8db-5e75-400f-b0af-5b617d4204a3-kube-api-access-bwmpf\") pod \"tempo-data-science-tempostack-querier-7f969fccfb-xbjdp\" (UID: \"2676a8db-5e75-400f-b0af-5b617d4204a3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp" Oct 31 00:33:38.380551 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.380499 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f26817af-cbdf-4baa-86b0-8b21f395779a-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h\" (UID: \"f26817af-cbdf-4baa-86b0-8b21f395779a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h" Oct 31 00:33:38.380551 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.380526 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/2676a8db-5e75-400f-b0af-5b617d4204a3-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-7f969fccfb-xbjdp\" (UID: \"2676a8db-5e75-400f-b0af-5b617d4204a3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp" Oct 31 00:33:38.380936 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.380559 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2676a8db-5e75-400f-b0af-5b617d4204a3-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-7f969fccfb-xbjdp\" (UID: \"2676a8db-5e75-400f-b0af-5b617d4204a3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp" Oct 31 00:33:38.380936 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.380596 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/f26817af-cbdf-4baa-86b0-8b21f395779a-storage-gcs-key\") pod \"tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h\" (UID: \"f26817af-cbdf-4baa-86b0-8b21f395779a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h" Oct 31 00:33:38.380936 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.380622 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/2676a8db-5e75-400f-b0af-5b617d4204a3-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-7f969fccfb-xbjdp\" (UID: \"2676a8db-5e75-400f-b0af-5b617d4204a3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp" Oct 31 00:33:38.380936 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.380666 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f26817af-cbdf-4baa-86b0-8b21f395779a-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h\" (UID: \"f26817af-cbdf-4baa-86b0-8b21f395779a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h" Oct 31 00:33:38.380936 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.380693 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/f26817af-cbdf-4baa-86b0-8b21f395779a-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h\" (UID: \"f26817af-cbdf-4baa-86b0-8b21f395779a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h" Oct 31 00:33:38.380936 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.380706 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/68837eb6-0f8f-4215-8771-1d661393310c-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8\" (UID: \"68837eb6-0f8f-4215-8771-1d661393310c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8" Oct 31 00:33:38.381371 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.381063 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68837eb6-0f8f-4215-8771-1d661393310c-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8\" (UID: \"68837eb6-0f8f-4215-8771-1d661393310c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8" Oct 31 00:33:38.381566 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.381542 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/68837eb6-0f8f-4215-8771-1d661393310c-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8\" (UID: \"68837eb6-0f8f-4215-8771-1d661393310c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8" Oct 31 00:33:38.382023 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.382003 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/2676a8db-5e75-400f-b0af-5b617d4204a3-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-7f969fccfb-xbjdp\" (UID: \"2676a8db-5e75-400f-b0af-5b617d4204a3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp" Oct 31 00:33:38.382183 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.382148 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2676a8db-5e75-400f-b0af-5b617d4204a3-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-7f969fccfb-xbjdp\" (UID: \"2676a8db-5e75-400f-b0af-5b617d4204a3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp" Oct 31 00:33:38.382525 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.382504 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/2676a8db-5e75-400f-b0af-5b617d4204a3-tempo-conf\") pod \"tempo-data-science-tempostack-querier-7f969fccfb-xbjdp\" (UID: \"2676a8db-5e75-400f-b0af-5b617d4204a3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp" Oct 31 00:33:38.383237 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.383135 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f26817af-cbdf-4baa-86b0-8b21f395779a-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h\" (UID: \"f26817af-cbdf-4baa-86b0-8b21f395779a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h" Oct 31 00:33:38.383803 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.383773 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f26817af-cbdf-4baa-86b0-8b21f395779a-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h\" (UID: \"f26817af-cbdf-4baa-86b0-8b21f395779a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h" Oct 31 00:33:38.383890 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.383809 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/68837eb6-0f8f-4215-8771-1d661393310c-storage-gcs-key\") pod \"tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8\" (UID: \"68837eb6-0f8f-4215-8771-1d661393310c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8" Oct 31 00:33:38.384219 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.384192 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/2676a8db-5e75-400f-b0af-5b617d4204a3-storage-gcs-key\") pod \"tempo-data-science-tempostack-querier-7f969fccfb-xbjdp\" (UID: \"2676a8db-5e75-400f-b0af-5b617d4204a3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp" Oct 31 00:33:38.384345 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.384320 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/68837eb6-0f8f-4215-8771-1d661393310c-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8\" (UID: \"68837eb6-0f8f-4215-8771-1d661393310c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8" Oct 31 00:33:38.384404 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.384339 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/f26817af-cbdf-4baa-86b0-8b21f395779a-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h\" (UID: \"f26817af-cbdf-4baa-86b0-8b21f395779a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h" Oct 31 00:33:38.384404 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.384339 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/2676a8db-5e75-400f-b0af-5b617d4204a3-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-7f969fccfb-xbjdp\" (UID: \"2676a8db-5e75-400f-b0af-5b617d4204a3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp" Oct 31 00:33:38.384597 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.384574 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/f26817af-cbdf-4baa-86b0-8b21f395779a-storage-gcs-key\") pod \"tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h\" (UID: \"f26817af-cbdf-4baa-86b0-8b21f395779a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h" Oct 31 00:33:38.396465 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.396408 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhkdj\" (UniqueName: \"kubernetes.io/projected/68837eb6-0f8f-4215-8771-1d661393310c-kube-api-access-zhkdj\") pod \"tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8\" (UID: \"68837eb6-0f8f-4215-8771-1d661393310c\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8" Oct 31 00:33:38.398111 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.398088 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwmpf\" (UniqueName: \"kubernetes.io/projected/2676a8db-5e75-400f-b0af-5b617d4204a3-kube-api-access-bwmpf\") pod \"tempo-data-science-tempostack-querier-7f969fccfb-xbjdp\" (UID: \"2676a8db-5e75-400f-b0af-5b617d4204a3\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp" Oct 31 00:33:38.398553 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.398535 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgz2l\" (UniqueName: \"kubernetes.io/projected/f26817af-cbdf-4baa-86b0-8b21f395779a-kube-api-access-vgz2l\") pod \"tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h\" (UID: \"f26817af-cbdf-4baa-86b0-8b21f395779a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h" Oct 31 00:33:38.456427 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.456398 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h" Oct 31 00:33:38.476303 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.476267 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp" Oct 31 00:33:38.509623 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.509536 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8" Oct 31 00:33:38.549140 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.549110 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-8469665bdf-d4mt9"] Oct 31 00:33:38.555212 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.555186 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-8469665bdf-d4mt9" Oct 31 00:33:38.569449 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.568399 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-distributor-mtls\"" Oct 31 00:33:38.570686 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.570093 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-8469665bdf-d4mt9"] Oct 31 00:33:38.686553 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.686422 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/5080ff17-a9b9-4a43-b6a1-70cc8dedd32d-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-8469665bdf-d4mt9\" (UID: \"5080ff17-a9b9-4a43-b6a1-70cc8dedd32d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-8469665bdf-d4mt9" Oct 31 00:33:38.686553 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.686503 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/5080ff17-a9b9-4a43-b6a1-70cc8dedd32d-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-8469665bdf-d4mt9\" (UID: \"5080ff17-a9b9-4a43-b6a1-70cc8dedd32d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-8469665bdf-d4mt9" Oct 31 00:33:38.686732 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.686552 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5080ff17-a9b9-4a43-b6a1-70cc8dedd32d-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-8469665bdf-d4mt9\" (UID: \"5080ff17-a9b9-4a43-b6a1-70cc8dedd32d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-8469665bdf-d4mt9" Oct 31 00:33:38.686732 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.686620 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcdfp\" (UniqueName: \"kubernetes.io/projected/5080ff17-a9b9-4a43-b6a1-70cc8dedd32d-kube-api-access-tcdfp\") pod \"tempo-data-science-tempostack-distributor-8469665bdf-d4mt9\" (UID: \"5080ff17-a9b9-4a43-b6a1-70cc8dedd32d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-8469665bdf-d4mt9" Oct 31 00:33:38.686732 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.686657 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/5080ff17-a9b9-4a43-b6a1-70cc8dedd32d-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-8469665bdf-d4mt9\" (UID: \"5080ff17-a9b9-4a43-b6a1-70cc8dedd32d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-8469665bdf-d4mt9" Oct 31 00:33:38.739980 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.739939 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h"] Oct 31 00:33:38.742114 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:33:38.742081 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf26817af_cbdf_4baa_86b0_8b21f395779a.slice/crio-ff0a011bc9030da47ed7fe12edf6cba911a8f63acd453444053071953d7eb936 WatchSource:0}: Error finding container ff0a011bc9030da47ed7fe12edf6cba911a8f63acd453444053071953d7eb936: Status 404 returned error can't find the container with id ff0a011bc9030da47ed7fe12edf6cba911a8f63acd453444053071953d7eb936 Oct 31 00:33:38.771710 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.771685 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp"] Oct 31 00:33:38.773997 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:33:38.773964 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2676a8db_5e75_400f_b0af_5b617d4204a3.slice/crio-b1e10faf3bd03f707a980b545ff28c6ba54214321b16035ebff4c7504d855ec0 WatchSource:0}: Error finding container b1e10faf3bd03f707a980b545ff28c6ba54214321b16035ebff4c7504d855ec0: Status 404 returned error can't find the container with id b1e10faf3bd03f707a980b545ff28c6ba54214321b16035ebff4c7504d855ec0 Oct 31 00:33:38.788289 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.788260 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/5080ff17-a9b9-4a43-b6a1-70cc8dedd32d-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-8469665bdf-d4mt9\" (UID: \"5080ff17-a9b9-4a43-b6a1-70cc8dedd32d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-8469665bdf-d4mt9" Oct 31 00:33:38.788381 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.788328 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/5080ff17-a9b9-4a43-b6a1-70cc8dedd32d-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-8469665bdf-d4mt9\" (UID: \"5080ff17-a9b9-4a43-b6a1-70cc8dedd32d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-8469665bdf-d4mt9" Oct 31 00:33:38.788461 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.788430 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5080ff17-a9b9-4a43-b6a1-70cc8dedd32d-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-8469665bdf-d4mt9\" (UID: \"5080ff17-a9b9-4a43-b6a1-70cc8dedd32d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-8469665bdf-d4mt9" Oct 31 00:33:38.788581 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.788492 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcdfp\" (UniqueName: \"kubernetes.io/projected/5080ff17-a9b9-4a43-b6a1-70cc8dedd32d-kube-api-access-tcdfp\") pod \"tempo-data-science-tempostack-distributor-8469665bdf-d4mt9\" (UID: \"5080ff17-a9b9-4a43-b6a1-70cc8dedd32d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-8469665bdf-d4mt9" Oct 31 00:33:38.788581 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.788516 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/5080ff17-a9b9-4a43-b6a1-70cc8dedd32d-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-8469665bdf-d4mt9\" (UID: \"5080ff17-a9b9-4a43-b6a1-70cc8dedd32d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-8469665bdf-d4mt9" Oct 31 00:33:38.789172 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.789150 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/5080ff17-a9b9-4a43-b6a1-70cc8dedd32d-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-8469665bdf-d4mt9\" (UID: \"5080ff17-a9b9-4a43-b6a1-70cc8dedd32d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-8469665bdf-d4mt9" Oct 31 00:33:38.789469 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.789445 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5080ff17-a9b9-4a43-b6a1-70cc8dedd32d-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-8469665bdf-d4mt9\" (UID: \"5080ff17-a9b9-4a43-b6a1-70cc8dedd32d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-8469665bdf-d4mt9" Oct 31 00:33:38.789582 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.789539 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/5080ff17-a9b9-4a43-b6a1-70cc8dedd32d-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-8469665bdf-d4mt9\" (UID: \"5080ff17-a9b9-4a43-b6a1-70cc8dedd32d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-8469665bdf-d4mt9" Oct 31 00:33:38.791460 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.791430 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/5080ff17-a9b9-4a43-b6a1-70cc8dedd32d-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-8469665bdf-d4mt9\" (UID: \"5080ff17-a9b9-4a43-b6a1-70cc8dedd32d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-8469665bdf-d4mt9" Oct 31 00:33:38.801816 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.801796 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcdfp\" (UniqueName: \"kubernetes.io/projected/5080ff17-a9b9-4a43-b6a1-70cc8dedd32d-kube-api-access-tcdfp\") pod \"tempo-data-science-tempostack-distributor-8469665bdf-d4mt9\" (UID: \"5080ff17-a9b9-4a43-b6a1-70cc8dedd32d\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-8469665bdf-d4mt9" Oct 31 00:33:38.812184 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.812165 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8"] Oct 31 00:33:38.830094 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:33:38.830072 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68837eb6_0f8f_4215_8771_1d661393310c.slice/crio-c918903486443575e1be7334b6ecc5ec78b4d575bb89b88f2cea87b98bf0458c WatchSource:0}: Error finding container c918903486443575e1be7334b6ecc5ec78b4d575bb89b88f2cea87b98bf0458c: Status 404 returned error can't find the container with id c918903486443575e1be7334b6ecc5ec78b4d575bb89b88f2cea87b98bf0458c Oct 31 00:33:38.879041 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.879020 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-8469665bdf-d4mt9" Oct 31 00:33:38.885929 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.885906 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h" event={"ID":"f26817af-cbdf-4baa-86b0-8b21f395779a","Type":"ContainerStarted","Data":"ff0a011bc9030da47ed7fe12edf6cba911a8f63acd453444053071953d7eb936"} Oct 31 00:33:38.888369 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.887941 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp" event={"ID":"2676a8db-5e75-400f-b0af-5b617d4204a3","Type":"ContainerStarted","Data":"a726191aa1ad517d9cf02f5f8824eb833896a304026c09b65f1efe5a67b1afeb"} Oct 31 00:33:38.888369 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.888000 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp" event={"ID":"2676a8db-5e75-400f-b0af-5b617d4204a3","Type":"ContainerStarted","Data":"b1e10faf3bd03f707a980b545ff28c6ba54214321b16035ebff4c7504d855ec0"} Oct 31 00:33:38.888369 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.888141 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp" Oct 31 00:33:38.889690 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.889663 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8" event={"ID":"68837eb6-0f8f-4215-8771-1d661393310c","Type":"ContainerStarted","Data":"c918903486443575e1be7334b6ecc5ec78b4d575bb89b88f2cea87b98bf0458c"} Oct 31 00:33:38.910146 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:38.910104 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp" podStartSLOduration=0.910090185 podStartE2EDuration="910.090185ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:38.908186429 +0000 UTC m=+2055.545005148" watchObservedRunningTime="2026-10-31 00:33:38.910090185 +0000 UTC m=+2055.546908900" Oct 31 00:33:39.155298 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:39.155263 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-8469665bdf-d4mt9"] Oct 31 00:33:39.157897 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:33:39.157861 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5080ff17_a9b9_4a43_b6a1_70cc8dedd32d.slice/crio-a6f02bbcb25ece18f9784722b2566bf59792eb12743518f0634cce6e34a61d3d WatchSource:0}: Error finding container a6f02bbcb25ece18f9784722b2566bf59792eb12743518f0634cce6e34a61d3d: Status 404 returned error can't find the container with id a6f02bbcb25ece18f9784722b2566bf59792eb12743518f0634cce6e34a61d3d Oct 31 00:33:39.896505 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:39.896468 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h" event={"ID":"f26817af-cbdf-4baa-86b0-8b21f395779a","Type":"ContainerStarted","Data":"a1c4003672873d8e8ccd0079a2fd22ce6de5c2679233ff31ff816d6fe635ea46"} Oct 31 00:33:39.896678 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:39.896534 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h" Oct 31 00:33:39.898148 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:39.898121 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-8469665bdf-d4mt9" event={"ID":"5080ff17-a9b9-4a43-b6a1-70cc8dedd32d","Type":"ContainerStarted","Data":"2907c0819986add472541751eff6c22477872cbc162c0acb87bfd147be28ba33"} Oct 31 00:33:39.898287 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:39.898163 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-8469665bdf-d4mt9" event={"ID":"5080ff17-a9b9-4a43-b6a1-70cc8dedd32d","Type":"ContainerStarted","Data":"a6f02bbcb25ece18f9784722b2566bf59792eb12743518f0634cce6e34a61d3d"} Oct 31 00:33:39.898287 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:39.898187 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-8469665bdf-d4mt9" Oct 31 00:33:39.899639 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:39.899619 5274 generic.go:356] "Generic (PLEG): container finished" podID="68837eb6-0f8f-4215-8771-1d661393310c" containerID="62675eac782e908c5ac8b9047e9b91442a9b5317dd0af42d2dce0f033c98b609" exitCode=1 Oct 31 00:33:39.899708 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:39.899674 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8" event={"ID":"68837eb6-0f8f-4215-8771-1d661393310c","Type":"ContainerDied","Data":"62675eac782e908c5ac8b9047e9b91442a9b5317dd0af42d2dce0f033c98b609"} Oct 31 00:33:39.899837 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:39.899822 5274 scope.go:122] "RemoveContainer" containerID="62675eac782e908c5ac8b9047e9b91442a9b5317dd0af42d2dce0f033c98b609" Oct 31 00:33:39.917919 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:39.917882 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h" podStartSLOduration=1.917869126 podStartE2EDuration="1.917869126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:39.917396533 +0000 UTC m=+2056.554215250" watchObservedRunningTime="2026-10-31 00:33:39.917869126 +0000 UTC m=+2056.554687842" Oct 31 00:33:39.940687 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:39.940635 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-8469665bdf-d4mt9" podStartSLOduration=1.940619867 podStartE2EDuration="1.940619867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:39.939639121 +0000 UTC m=+2056.576457838" watchObservedRunningTime="2026-10-31 00:33:39.940619867 +0000 UTC m=+2056.577438583" Oct 31 00:33:40.906133 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:40.906092 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8" event={"ID":"68837eb6-0f8f-4215-8771-1d661393310c","Type":"ContainerStarted","Data":"a223d694c5c64ac2e502f1fc0d016c493be19468428a91ed1e217d0b35d44dcf"} Oct 31 00:33:40.906633 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:40.906613 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8" Oct 31 00:33:40.929060 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:40.929013 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8" podStartSLOduration=2.928998531 podStartE2EDuration="2.928998531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:40.927281805 +0000 UTC m=+2057.564100515" watchObservedRunningTime="2026-10-31 00:33:40.928998531 +0000 UTC m=+2057.565817250" Oct 31 00:33:46.875688 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:46.875655 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-q62ss" Oct 31 00:33:50.899388 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:33:50.899351 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:33:59.906021 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:33:59.905990 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp" Oct 31 00:34:00.912023 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:00.911992 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-8469665bdf-d4mt9" Oct 31 00:34:00.912737 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:00.912717 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h" Oct 31 00:34:01.111165 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:01.111127 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp"] Oct 31 00:34:01.111512 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:01.111320 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp" podUID="2676a8db-5e75-400f-b0af-5b617d4204a3" containerName="tempo" containerID="cri-o://a726191aa1ad517d9cf02f5f8824eb833896a304026c09b65f1efe5a67b1afeb" gracePeriod=30 Oct 31 00:34:01.219945 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:01.219865 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h"] Oct 31 00:34:01.220098 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:01.220061 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h" podUID="f26817af-cbdf-4baa-86b0-8b21f395779a" containerName="tempo" containerID="cri-o://a1c4003672873d8e8ccd0079a2fd22ce6de5c2679233ff31ff816d6fe635ea46" gracePeriod=30 Oct 31 00:34:02.922621 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:02.922570 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8" podUID="68837eb6-0f8f-4215-8771-1d661393310c" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:34:09.904090 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:09.904052 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp" podUID="2676a8db-5e75-400f-b0af-5b617d4204a3" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:34:11.050516 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:11.050475 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h" podUID="f26817af-cbdf-4baa-86b0-8b21f395779a" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:34:12.920211 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:12.920174 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8" podUID="68837eb6-0f8f-4215-8771-1d661393310c" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:34:19.903679 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:19.903641 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp" podUID="2676a8db-5e75-400f-b0af-5b617d4204a3" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:34:20.899756 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:34:20.899720 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:34:21.149710 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:21.149664 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h" podUID="f26817af-cbdf-4baa-86b0-8b21f395779a" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:34:22.921071 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:22.921033 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8" podUID="68837eb6-0f8f-4215-8771-1d661393310c" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:34:29.903680 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:29.903645 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp" podUID="2676a8db-5e75-400f-b0af-5b617d4204a3" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:34:29.904179 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:29.903745 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp" Oct 31 00:34:30.949689 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:30.949651 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h" podUID="f26817af-cbdf-4baa-86b0-8b21f395779a" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:34:30.950150 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:30.949771 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h" Oct 31 00:34:31.158087 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:31.158050 5274 generic.go:356] "Generic (PLEG): container finished" podID="2676a8db-5e75-400f-b0af-5b617d4204a3" containerID="a726191aa1ad517d9cf02f5f8824eb833896a304026c09b65f1efe5a67b1afeb" exitCode=137 Oct 31 00:34:31.158241 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:31.158120 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp" event={"ID":"2676a8db-5e75-400f-b0af-5b617d4204a3","Type":"ContainerDied","Data":"a726191aa1ad517d9cf02f5f8824eb833896a304026c09b65f1efe5a67b1afeb"} Oct 31 00:34:31.227970 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:34:31.227930 5274 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf26817af_cbdf_4baa_86b0_8b21f395779a.slice/crio-ff0a011bc9030da47ed7fe12edf6cba911a8f63acd453444053071953d7eb936\": RecentStats: unable to find data in memory cache]" Oct 31 00:34:31.356902 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:31.356876 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp" Oct 31 00:34:31.425225 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:31.425196 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2676a8db-5e75-400f-b0af-5b617d4204a3-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/2676a8db-5e75-400f-b0af-5b617d4204a3-tempo-conf\") pod \"2676a8db-5e75-400f-b0af-5b617d4204a3\" (UID: \"2676a8db-5e75-400f-b0af-5b617d4204a3\") " Oct 31 00:34:31.425382 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:31.425239 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2676a8db-5e75-400f-b0af-5b617d4204a3-kube-api-access-bwmpf\" (UniqueName: \"kubernetes.io/projected/2676a8db-5e75-400f-b0af-5b617d4204a3-kube-api-access-bwmpf\") pod \"2676a8db-5e75-400f-b0af-5b617d4204a3\" (UID: \"2676a8db-5e75-400f-b0af-5b617d4204a3\") " Oct 31 00:34:31.425382 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:31.425303 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2676a8db-5e75-400f-b0af-5b617d4204a3-tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/2676a8db-5e75-400f-b0af-5b617d4204a3-tempo-data-science-tempostack-querier-mtls\") pod \"2676a8db-5e75-400f-b0af-5b617d4204a3\" (UID: \"2676a8db-5e75-400f-b0af-5b617d4204a3\") " Oct 31 00:34:31.425382 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:31.425322 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2676a8db-5e75-400f-b0af-5b617d4204a3-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2676a8db-5e75-400f-b0af-5b617d4204a3-tempo-data-science-tempostack-ca-bundle\") pod \"2676a8db-5e75-400f-b0af-5b617d4204a3\" (UID: \"2676a8db-5e75-400f-b0af-5b617d4204a3\") " Oct 31 00:34:31.425382 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:31.425358 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2676a8db-5e75-400f-b0af-5b617d4204a3-storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/2676a8db-5e75-400f-b0af-5b617d4204a3-storage-gcs-key\") pod \"2676a8db-5e75-400f-b0af-5b617d4204a3\" (UID: \"2676a8db-5e75-400f-b0af-5b617d4204a3\") " Oct 31 00:34:31.425382 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:31.425379 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2676a8db-5e75-400f-b0af-5b617d4204a3-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/2676a8db-5e75-400f-b0af-5b617d4204a3-tempo-tmp-storage\") pod \"2676a8db-5e75-400f-b0af-5b617d4204a3\" (UID: \"2676a8db-5e75-400f-b0af-5b617d4204a3\") " Oct 31 00:34:31.425749 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:31.425725 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2676a8db-5e75-400f-b0af-5b617d4204a3-tempo-conf" pod "2676a8db-5e75-400f-b0af-5b617d4204a3" (UID: "2676a8db-5e75-400f-b0af-5b617d4204a3"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:34:31.425842 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:31.425743 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2676a8db-5e75-400f-b0af-5b617d4204a3-tempo-data-science-tempostack-ca-bundle" pod "2676a8db-5e75-400f-b0af-5b617d4204a3" (UID: "2676a8db-5e75-400f-b0af-5b617d4204a3"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:34:31.425842 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:31.425757 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2676a8db-5e75-400f-b0af-5b617d4204a3-tempo-tmp-storage" pod "2676a8db-5e75-400f-b0af-5b617d4204a3" (UID: "2676a8db-5e75-400f-b0af-5b617d4204a3"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:34:31.427610 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:31.427586 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2676a8db-5e75-400f-b0af-5b617d4204a3-kube-api-access-bwmpf" pod "2676a8db-5e75-400f-b0af-5b617d4204a3" (UID: "2676a8db-5e75-400f-b0af-5b617d4204a3"). InnerVolumeSpecName "kube-api-access-bwmpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:34:31.427610 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:31.427600 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2676a8db-5e75-400f-b0af-5b617d4204a3-tempo-data-science-tempostack-querier-mtls" pod "2676a8db-5e75-400f-b0af-5b617d4204a3" (UID: "2676a8db-5e75-400f-b0af-5b617d4204a3"). InnerVolumeSpecName "tempo-data-science-tempostack-querier-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:34:31.427802 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:31.427620 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2676a8db-5e75-400f-b0af-5b617d4204a3-storage-gcs-key" pod "2676a8db-5e75-400f-b0af-5b617d4204a3" (UID: "2676a8db-5e75-400f-b0af-5b617d4204a3"). InnerVolumeSpecName "storage-gcs-key". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:34:31.526664 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:31.526642 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/2676a8db-5e75-400f-b0af-5b617d4204a3-tempo-conf\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:31.526664 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:31.526664 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwmpf\" (UniqueName: \"kubernetes.io/projected/2676a8db-5e75-400f-b0af-5b617d4204a3-kube-api-access-bwmpf\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:31.526824 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:31.526675 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/2676a8db-5e75-400f-b0af-5b617d4204a3-tempo-data-science-tempostack-querier-mtls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:31.526824 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:31.526685 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2676a8db-5e75-400f-b0af-5b617d4204a3-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:31.526824 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:31.526695 5274 reconciler_common.go:299] "Volume detached for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/2676a8db-5e75-400f-b0af-5b617d4204a3-storage-gcs-key\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:31.526824 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:31.526703 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/2676a8db-5e75-400f-b0af-5b617d4204a3-tempo-tmp-storage\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:31.855901 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:31.855877 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h" Oct 31 00:34:31.930230 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:31.930199 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f26817af-cbdf-4baa-86b0-8b21f395779a-tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/f26817af-cbdf-4baa-86b0-8b21f395779a-tempo-data-science-tempostack-query-frontend-mtls\") pod \"f26817af-cbdf-4baa-86b0-8b21f395779a\" (UID: \"f26817af-cbdf-4baa-86b0-8b21f395779a\") " Oct 31 00:34:31.930401 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:31.930263 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f26817af-cbdf-4baa-86b0-8b21f395779a-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f26817af-cbdf-4baa-86b0-8b21f395779a-tempo-conf\") pod \"f26817af-cbdf-4baa-86b0-8b21f395779a\" (UID: \"f26817af-cbdf-4baa-86b0-8b21f395779a\") " Oct 31 00:34:31.930401 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:31.930312 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26817af-cbdf-4baa-86b0-8b21f395779a-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/f26817af-cbdf-4baa-86b0-8b21f395779a-tempo-tmp-storage\") pod \"f26817af-cbdf-4baa-86b0-8b21f395779a\" (UID: \"f26817af-cbdf-4baa-86b0-8b21f395779a\") " Oct 31 00:34:31.930401 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:31.930368 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f26817af-cbdf-4baa-86b0-8b21f395779a-storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/f26817af-cbdf-4baa-86b0-8b21f395779a-storage-gcs-key\") pod \"f26817af-cbdf-4baa-86b0-8b21f395779a\" (UID: \"f26817af-cbdf-4baa-86b0-8b21f395779a\") " Oct 31 00:34:31.930401 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:31.930393 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f26817af-cbdf-4baa-86b0-8b21f395779a-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f26817af-cbdf-4baa-86b0-8b21f395779a-tempo-data-science-tempostack-ca-bundle\") pod \"f26817af-cbdf-4baa-86b0-8b21f395779a\" (UID: \"f26817af-cbdf-4baa-86b0-8b21f395779a\") " Oct 31 00:34:31.930606 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:31.930449 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f26817af-cbdf-4baa-86b0-8b21f395779a-kube-api-access-vgz2l\" (UniqueName: \"kubernetes.io/projected/f26817af-cbdf-4baa-86b0-8b21f395779a-kube-api-access-vgz2l\") pod \"f26817af-cbdf-4baa-86b0-8b21f395779a\" (UID: \"f26817af-cbdf-4baa-86b0-8b21f395779a\") " Oct 31 00:34:31.930761 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:31.930733 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26817af-cbdf-4baa-86b0-8b21f395779a-tempo-tmp-storage" pod "f26817af-cbdf-4baa-86b0-8b21f395779a" (UID: "f26817af-cbdf-4baa-86b0-8b21f395779a"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:34:31.930833 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:31.930772 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f26817af-cbdf-4baa-86b0-8b21f395779a-tempo-conf" pod "f26817af-cbdf-4baa-86b0-8b21f395779a" (UID: "f26817af-cbdf-4baa-86b0-8b21f395779a"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:34:31.930833 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:31.930767 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f26817af-cbdf-4baa-86b0-8b21f395779a-tempo-data-science-tempostack-ca-bundle" pod "f26817af-cbdf-4baa-86b0-8b21f395779a" (UID: "f26817af-cbdf-4baa-86b0-8b21f395779a"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:34:31.932396 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:31.932370 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f26817af-cbdf-4baa-86b0-8b21f395779a-tempo-data-science-tempostack-query-frontend-mtls" pod "f26817af-cbdf-4baa-86b0-8b21f395779a" (UID: "f26817af-cbdf-4baa-86b0-8b21f395779a"). InnerVolumeSpecName "tempo-data-science-tempostack-query-frontend-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:34:31.932494 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:31.932466 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f26817af-cbdf-4baa-86b0-8b21f395779a-storage-gcs-key" pod "f26817af-cbdf-4baa-86b0-8b21f395779a" (UID: "f26817af-cbdf-4baa-86b0-8b21f395779a"). InnerVolumeSpecName "storage-gcs-key". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:34:31.932554 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:31.932531 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f26817af-cbdf-4baa-86b0-8b21f395779a-kube-api-access-vgz2l" pod "f26817af-cbdf-4baa-86b0-8b21f395779a" (UID: "f26817af-cbdf-4baa-86b0-8b21f395779a"). InnerVolumeSpecName "kube-api-access-vgz2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:34:32.031666 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:32.031600 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/f26817af-cbdf-4baa-86b0-8b21f395779a-tempo-data-science-tempostack-query-frontend-mtls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:32.031666 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:32.031627 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f26817af-cbdf-4baa-86b0-8b21f395779a-tempo-conf\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:32.031666 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:32.031638 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/f26817af-cbdf-4baa-86b0-8b21f395779a-tempo-tmp-storage\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:32.031666 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:32.031648 5274 reconciler_common.go:299] "Volume detached for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/f26817af-cbdf-4baa-86b0-8b21f395779a-storage-gcs-key\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:32.031666 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:32.031658 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f26817af-cbdf-4baa-86b0-8b21f395779a-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:32.031666 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:32.031667 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgz2l\" (UniqueName: \"kubernetes.io/projected/f26817af-cbdf-4baa-86b0-8b21f395779a-kube-api-access-vgz2l\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:32.165192 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:32.165161 5274 generic.go:356] "Generic (PLEG): container finished" podID="f26817af-cbdf-4baa-86b0-8b21f395779a" containerID="a1c4003672873d8e8ccd0079a2fd22ce6de5c2679233ff31ff816d6fe635ea46" exitCode=137 Oct 31 00:34:32.165311 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:32.165205 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h" Oct 31 00:34:32.165311 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:32.165240 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h" event={"ID":"f26817af-cbdf-4baa-86b0-8b21f395779a","Type":"ContainerDied","Data":"a1c4003672873d8e8ccd0079a2fd22ce6de5c2679233ff31ff816d6fe635ea46"} Oct 31 00:34:32.165311 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:32.165290 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h" event={"ID":"f26817af-cbdf-4baa-86b0-8b21f395779a","Type":"ContainerDied","Data":"ff0a011bc9030da47ed7fe12edf6cba911a8f63acd453444053071953d7eb936"} Oct 31 00:34:32.165439 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:32.165313 5274 scope.go:122] "RemoveContainer" containerID="a1c4003672873d8e8ccd0079a2fd22ce6de5c2679233ff31ff816d6fe635ea46" Oct 31 00:34:32.166593 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:32.166571 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp" event={"ID":"2676a8db-5e75-400f-b0af-5b617d4204a3","Type":"ContainerDied","Data":"b1e10faf3bd03f707a980b545ff28c6ba54214321b16035ebff4c7504d855ec0"} Oct 31 00:34:32.166670 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:32.166626 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp" Oct 31 00:34:32.174777 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:32.174758 5274 scope.go:122] "RemoveContainer" containerID="a1c4003672873d8e8ccd0079a2fd22ce6de5c2679233ff31ff816d6fe635ea46" Oct 31 00:34:32.175366 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:34:32.175341 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1c4003672873d8e8ccd0079a2fd22ce6de5c2679233ff31ff816d6fe635ea46\": container with ID starting with a1c4003672873d8e8ccd0079a2fd22ce6de5c2679233ff31ff816d6fe635ea46 not found: ID does not exist" containerID="a1c4003672873d8e8ccd0079a2fd22ce6de5c2679233ff31ff816d6fe635ea46" Oct 31 00:34:32.175458 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:32.175372 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1c4003672873d8e8ccd0079a2fd22ce6de5c2679233ff31ff816d6fe635ea46"} err="failed to get container status \"a1c4003672873d8e8ccd0079a2fd22ce6de5c2679233ff31ff816d6fe635ea46\": rpc error: code = NotFound desc = could not find container \"a1c4003672873d8e8ccd0079a2fd22ce6de5c2679233ff31ff816d6fe635ea46\": container with ID starting with a1c4003672873d8e8ccd0079a2fd22ce6de5c2679233ff31ff816d6fe635ea46 not found: ID does not exist" Oct 31 00:34:32.175458 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:32.175389 5274 scope.go:122] "RemoveContainer" containerID="a726191aa1ad517d9cf02f5f8824eb833896a304026c09b65f1efe5a67b1afeb" Oct 31 00:34:32.194486 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:32.194466 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp"] Oct 31 00:34:32.198854 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:32.198835 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-7f969fccfb-xbjdp"] Oct 31 00:34:32.211630 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:32.211609 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h"] Oct 31 00:34:32.216216 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:32.216197 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-8b6b8c89f-68n2h"] Oct 31 00:34:32.920678 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:32.920645 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8" podUID="68837eb6-0f8f-4215-8771-1d661393310c" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:34:33.481525 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:33.481496 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2676a8db-5e75-400f-b0af-5b617d4204a3" path="/var/lib/kubelet/pods/2676a8db-5e75-400f-b0af-5b617d4204a3/volumes" Oct 31 00:34:33.481911 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:33.481895 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f26817af-cbdf-4baa-86b0-8b21f395779a" path="/var/lib/kubelet/pods/f26817af-cbdf-4baa-86b0-8b21f395779a/volumes" Oct 31 00:34:42.920883 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:42.920844 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8" podUID="68837eb6-0f8f-4215-8771-1d661393310c" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:34:49.488234 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:49.483158 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8" Oct 31 00:34:50.898785 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:34:50.898750 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:34:51.105918 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:51.105888 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8"] Oct 31 00:34:51.260499 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:51.260388 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8" podUID="68837eb6-0f8f-4215-8771-1d661393310c" containerName="tempo" containerID="cri-o://a223d694c5c64ac2e502f1fc0d016c493be19468428a91ed1e217d0b35d44dcf" gracePeriod=30 Oct 31 00:34:52.444327 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:52.444201 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:34:52.445538 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:52.444477 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="cb91015f-7854-47ed-9d1b-b81a71661356" containerName="otc-container" containerID="cri-o://c4079bca86cd83bcfc1c13cbc6ce9727738aa31d43f18d7c03aabbe79b8cacea" gracePeriod=30 Oct 31 00:34:52.569303 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:52.569282 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8" Oct 31 00:34:52.690366 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:52.690341 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:34:52.693297 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:52.693276 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68837eb6-0f8f-4215-8771-1d661393310c-kube-api-access-zhkdj\" (UniqueName: \"kubernetes.io/projected/68837eb6-0f8f-4215-8771-1d661393310c-kube-api-access-zhkdj\") pod \"68837eb6-0f8f-4215-8771-1d661393310c\" (UID: \"68837eb6-0f8f-4215-8771-1d661393310c\") " Oct 31 00:34:52.693430 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:52.693313 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68837eb6-0f8f-4215-8771-1d661393310c-storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/68837eb6-0f8f-4215-8771-1d661393310c-storage-gcs-key\") pod \"68837eb6-0f8f-4215-8771-1d661393310c\" (UID: \"68837eb6-0f8f-4215-8771-1d661393310c\") " Oct 31 00:34:52.693430 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:52.693366 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68837eb6-0f8f-4215-8771-1d661393310c-tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/68837eb6-0f8f-4215-8771-1d661393310c-tempo-data-science-tempostack-compactor-mtls\") pod \"68837eb6-0f8f-4215-8771-1d661393310c\" (UID: \"68837eb6-0f8f-4215-8771-1d661393310c\") " Oct 31 00:34:52.693554 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:52.693505 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68837eb6-0f8f-4215-8771-1d661393310c-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68837eb6-0f8f-4215-8771-1d661393310c-tempo-data-science-tempostack-ca-bundle\") pod \"68837eb6-0f8f-4215-8771-1d661393310c\" (UID: \"68837eb6-0f8f-4215-8771-1d661393310c\") " Oct 31 00:34:52.693613 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:52.693567 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68837eb6-0f8f-4215-8771-1d661393310c-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/68837eb6-0f8f-4215-8771-1d661393310c-tempo-tmp-storage\") pod \"68837eb6-0f8f-4215-8771-1d661393310c\" (UID: \"68837eb6-0f8f-4215-8771-1d661393310c\") " Oct 31 00:34:52.693613 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:52.693596 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68837eb6-0f8f-4215-8771-1d661393310c-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/68837eb6-0f8f-4215-8771-1d661393310c-tempo-conf\") pod \"68837eb6-0f8f-4215-8771-1d661393310c\" (UID: \"68837eb6-0f8f-4215-8771-1d661393310c\") " Oct 31 00:34:52.694062 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:52.693946 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68837eb6-0f8f-4215-8771-1d661393310c-tempo-data-science-tempostack-ca-bundle" pod "68837eb6-0f8f-4215-8771-1d661393310c" (UID: "68837eb6-0f8f-4215-8771-1d661393310c"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:34:52.694289 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:52.694270 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68837eb6-0f8f-4215-8771-1d661393310c-tempo-tmp-storage" pod "68837eb6-0f8f-4215-8771-1d661393310c" (UID: "68837eb6-0f8f-4215-8771-1d661393310c"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:34:52.694734 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:52.694715 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68837eb6-0f8f-4215-8771-1d661393310c-tempo-conf" pod "68837eb6-0f8f-4215-8771-1d661393310c" (UID: "68837eb6-0f8f-4215-8771-1d661393310c"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:34:52.695457 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:52.695439 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68837eb6-0f8f-4215-8771-1d661393310c-kube-api-access-zhkdj" pod "68837eb6-0f8f-4215-8771-1d661393310c" (UID: "68837eb6-0f8f-4215-8771-1d661393310c"). InnerVolumeSpecName "kube-api-access-zhkdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:34:52.695831 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:52.695811 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68837eb6-0f8f-4215-8771-1d661393310c-storage-gcs-key" pod "68837eb6-0f8f-4215-8771-1d661393310c" (UID: "68837eb6-0f8f-4215-8771-1d661393310c"). InnerVolumeSpecName "storage-gcs-key". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:34:52.695916 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:52.695857 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68837eb6-0f8f-4215-8771-1d661393310c-tempo-data-science-tempostack-compactor-mtls" pod "68837eb6-0f8f-4215-8771-1d661393310c" (UID: "68837eb6-0f8f-4215-8771-1d661393310c"). InnerVolumeSpecName "tempo-data-science-tempostack-compactor-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:34:52.794737 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:52.794705 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb91015f-7854-47ed-9d1b-b81a71661356-kube-api-access-tbmlm\" (UniqueName: \"kubernetes.io/projected/cb91015f-7854-47ed-9d1b-b81a71661356-kube-api-access-tbmlm\") pod \"cb91015f-7854-47ed-9d1b-b81a71661356\" (UID: \"cb91015f-7854-47ed-9d1b-b81a71661356\") " Oct 31 00:34:52.794918 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:52.794763 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb91015f-7854-47ed-9d1b-b81a71661356-otc-internal\" (UniqueName: \"kubernetes.io/configmap/cb91015f-7854-47ed-9d1b-b81a71661356-otc-internal\") pod \"cb91015f-7854-47ed-9d1b-b81a71661356\" (UID: \"cb91015f-7854-47ed-9d1b-b81a71661356\") " Oct 31 00:34:52.794918 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:52.794803 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb91015f-7854-47ed-9d1b-b81a71661356-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb91015f-7854-47ed-9d1b-b81a71661356-otlp-tls-certs\") pod \"cb91015f-7854-47ed-9d1b-b81a71661356\" (UID: \"cb91015f-7854-47ed-9d1b-b81a71661356\") " Oct 31 00:34:52.794918 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:52.794864 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb91015f-7854-47ed-9d1b-b81a71661356-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb91015f-7854-47ed-9d1b-b81a71661356-monitoring-tls-certs\") pod \"cb91015f-7854-47ed-9d1b-b81a71661356\" (UID: \"cb91015f-7854-47ed-9d1b-b81a71661356\") " Oct 31 00:34:52.795151 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:52.795133 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhkdj\" (UniqueName: \"kubernetes.io/projected/68837eb6-0f8f-4215-8771-1d661393310c-kube-api-access-zhkdj\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:52.795213 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:52.795155 5274 reconciler_common.go:299] "Volume detached for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/68837eb6-0f8f-4215-8771-1d661393310c-storage-gcs-key\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:52.795213 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:52.795169 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/68837eb6-0f8f-4215-8771-1d661393310c-tempo-data-science-tempostack-compactor-mtls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:52.795213 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:52.795179 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68837eb6-0f8f-4215-8771-1d661393310c-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:52.795213 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:52.795190 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/68837eb6-0f8f-4215-8771-1d661393310c-tempo-tmp-storage\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:52.795213 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:52.795200 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/68837eb6-0f8f-4215-8771-1d661393310c-tempo-conf\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:52.795378 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:52.795208 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb91015f-7854-47ed-9d1b-b81a71661356-otc-internal" pod "cb91015f-7854-47ed-9d1b-b81a71661356" (UID: "cb91015f-7854-47ed-9d1b-b81a71661356"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:34:52.796885 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:52.796857 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb91015f-7854-47ed-9d1b-b81a71661356-monitoring-tls-certs" pod "cb91015f-7854-47ed-9d1b-b81a71661356" (UID: "cb91015f-7854-47ed-9d1b-b81a71661356"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:34:52.797012 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:52.796891 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb91015f-7854-47ed-9d1b-b81a71661356-otlp-tls-certs" pod "cb91015f-7854-47ed-9d1b-b81a71661356" (UID: "cb91015f-7854-47ed-9d1b-b81a71661356"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:34:52.797012 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:52.796907 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb91015f-7854-47ed-9d1b-b81a71661356-kube-api-access-tbmlm" pod "cb91015f-7854-47ed-9d1b-b81a71661356" (UID: "cb91015f-7854-47ed-9d1b-b81a71661356"). InnerVolumeSpecName "kube-api-access-tbmlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:34:52.896363 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:52.896305 5274 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cb91015f-7854-47ed-9d1b-b81a71661356-otc-internal\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:52.896363 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:52.896325 5274 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb91015f-7854-47ed-9d1b-b81a71661356-otlp-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:52.896363 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:52.896334 5274 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb91015f-7854-47ed-9d1b-b81a71661356-monitoring-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:52.896363 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:52.896345 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbmlm\" (UniqueName: \"kubernetes.io/projected/cb91015f-7854-47ed-9d1b-b81a71661356-kube-api-access-tbmlm\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:53.270984 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.270881 5274 generic.go:356] "Generic (PLEG): container finished" podID="68837eb6-0f8f-4215-8771-1d661393310c" containerID="a223d694c5c64ac2e502f1fc0d016c493be19468428a91ed1e217d0b35d44dcf" exitCode=0 Oct 31 00:34:53.270984 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.270935 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8" Oct 31 00:34:53.271202 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.270976 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8" event={"ID":"68837eb6-0f8f-4215-8771-1d661393310c","Type":"ContainerDied","Data":"a223d694c5c64ac2e502f1fc0d016c493be19468428a91ed1e217d0b35d44dcf"} Oct 31 00:34:53.271202 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.271035 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8" event={"ID":"68837eb6-0f8f-4215-8771-1d661393310c","Type":"ContainerDied","Data":"c918903486443575e1be7334b6ecc5ec78b4d575bb89b88f2cea87b98bf0458c"} Oct 31 00:34:53.271202 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.271069 5274 scope.go:122] "RemoveContainer" containerID="a223d694c5c64ac2e502f1fc0d016c493be19468428a91ed1e217d0b35d44dcf" Oct 31 00:34:53.272387 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.272286 5274 generic.go:356] "Generic (PLEG): container finished" podID="cb91015f-7854-47ed-9d1b-b81a71661356" containerID="c4079bca86cd83bcfc1c13cbc6ce9727738aa31d43f18d7c03aabbe79b8cacea" exitCode=0 Oct 31 00:34:53.272387 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.272346 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:34:53.272517 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.272364 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"cb91015f-7854-47ed-9d1b-b81a71661356","Type":"ContainerDied","Data":"c4079bca86cd83bcfc1c13cbc6ce9727738aa31d43f18d7c03aabbe79b8cacea"} Oct 31 00:34:53.272517 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.272427 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"cb91015f-7854-47ed-9d1b-b81a71661356","Type":"ContainerDied","Data":"83d50264cb3a1667ad372b036198925d03f382d8b0a62679d9a373a0a5f52b48"} Oct 31 00:34:53.284544 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.284529 5274 scope.go:122] "RemoveContainer" containerID="62675eac782e908c5ac8b9047e9b91442a9b5317dd0af42d2dce0f033c98b609" Oct 31 00:34:53.292473 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.292453 5274 scope.go:122] "RemoveContainer" containerID="a223d694c5c64ac2e502f1fc0d016c493be19468428a91ed1e217d0b35d44dcf" Oct 31 00:34:53.292718 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:34:53.292696 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a223d694c5c64ac2e502f1fc0d016c493be19468428a91ed1e217d0b35d44dcf\": container with ID starting with a223d694c5c64ac2e502f1fc0d016c493be19468428a91ed1e217d0b35d44dcf not found: ID does not exist" containerID="a223d694c5c64ac2e502f1fc0d016c493be19468428a91ed1e217d0b35d44dcf" Oct 31 00:34:53.292767 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.292739 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a223d694c5c64ac2e502f1fc0d016c493be19468428a91ed1e217d0b35d44dcf"} err="failed to get container status \"a223d694c5c64ac2e502f1fc0d016c493be19468428a91ed1e217d0b35d44dcf\": rpc error: code = NotFound desc = could not find container \"a223d694c5c64ac2e502f1fc0d016c493be19468428a91ed1e217d0b35d44dcf\": container with ID starting with a223d694c5c64ac2e502f1fc0d016c493be19468428a91ed1e217d0b35d44dcf not found: ID does not exist" Oct 31 00:34:53.292767 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.292756 5274 scope.go:122] "RemoveContainer" containerID="62675eac782e908c5ac8b9047e9b91442a9b5317dd0af42d2dce0f033c98b609" Oct 31 00:34:53.292983 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:34:53.292949 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62675eac782e908c5ac8b9047e9b91442a9b5317dd0af42d2dce0f033c98b609\": container with ID starting with 62675eac782e908c5ac8b9047e9b91442a9b5317dd0af42d2dce0f033c98b609 not found: ID does not exist" containerID="62675eac782e908c5ac8b9047e9b91442a9b5317dd0af42d2dce0f033c98b609" Oct 31 00:34:53.293022 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.292989 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62675eac782e908c5ac8b9047e9b91442a9b5317dd0af42d2dce0f033c98b609"} err="failed to get container status \"62675eac782e908c5ac8b9047e9b91442a9b5317dd0af42d2dce0f033c98b609\": rpc error: code = NotFound desc = could not find container \"62675eac782e908c5ac8b9047e9b91442a9b5317dd0af42d2dce0f033c98b609\": container with ID starting with 62675eac782e908c5ac8b9047e9b91442a9b5317dd0af42d2dce0f033c98b609 not found: ID does not exist" Oct 31 00:34:53.293022 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.293003 5274 scope.go:122] "RemoveContainer" containerID="c4079bca86cd83bcfc1c13cbc6ce9727738aa31d43f18d7c03aabbe79b8cacea" Oct 31 00:34:53.301074 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.301053 5274 scope.go:122] "RemoveContainer" containerID="c4079bca86cd83bcfc1c13cbc6ce9727738aa31d43f18d7c03aabbe79b8cacea" Oct 31 00:34:53.301339 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:34:53.301319 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4079bca86cd83bcfc1c13cbc6ce9727738aa31d43f18d7c03aabbe79b8cacea\": container with ID starting with c4079bca86cd83bcfc1c13cbc6ce9727738aa31d43f18d7c03aabbe79b8cacea not found: ID does not exist" containerID="c4079bca86cd83bcfc1c13cbc6ce9727738aa31d43f18d7c03aabbe79b8cacea" Oct 31 00:34:53.301441 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.301345 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4079bca86cd83bcfc1c13cbc6ce9727738aa31d43f18d7c03aabbe79b8cacea"} err="failed to get container status \"c4079bca86cd83bcfc1c13cbc6ce9727738aa31d43f18d7c03aabbe79b8cacea\": rpc error: code = NotFound desc = could not find container \"c4079bca86cd83bcfc1c13cbc6ce9727738aa31d43f18d7c03aabbe79b8cacea\": container with ID starting with c4079bca86cd83bcfc1c13cbc6ce9727738aa31d43f18d7c03aabbe79b8cacea not found: ID does not exist" Oct 31 00:34:53.306814 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.304409 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8"] Oct 31 00:34:53.311007 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.310988 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-68b748f7bd-qmtk8"] Oct 31 00:34:53.326074 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.326054 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:34:53.332058 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.332037 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:34:53.367541 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.367519 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:34:53.368143 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.368104 5274 cpu_manager.go:414] "containerMap: removing container" podUID="68837eb6-0f8f-4215-8771-1d661393310c" containerName="tempo" Oct 31 00:34:53.368307 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.368146 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68837eb6-0f8f-4215-8771-1d661393310c" containerName="tempo" Oct 31 00:34:53.368307 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.368165 5274 cpu_manager.go:414] "containerMap: removing container" podUID="cb91015f-7854-47ed-9d1b-b81a71661356" containerName="otc-container" Oct 31 00:34:53.368307 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.368173 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb91015f-7854-47ed-9d1b-b81a71661356" containerName="otc-container" Oct 31 00:34:53.368307 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.368216 5274 cpu_manager.go:414] "containerMap: removing container" podUID="2676a8db-5e75-400f-b0af-5b617d4204a3" containerName="tempo" Oct 31 00:34:53.368307 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.368225 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2676a8db-5e75-400f-b0af-5b617d4204a3" containerName="tempo" Oct 31 00:34:53.368307 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.368244 5274 cpu_manager.go:414] "containerMap: removing container" podUID="f26817af-cbdf-4baa-86b0-8b21f395779a" containerName="tempo" Oct 31 00:34:53.368307 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.368251 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f26817af-cbdf-4baa-86b0-8b21f395779a" containerName="tempo" Oct 31 00:34:53.368307 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.368260 5274 cpu_manager.go:414] "containerMap: removing container" podUID="68837eb6-0f8f-4215-8771-1d661393310c" containerName="tempo" Oct 31 00:34:53.368307 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.368267 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68837eb6-0f8f-4215-8771-1d661393310c" containerName="tempo" Oct 31 00:34:53.368769 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.368375 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="2676a8db-5e75-400f-b0af-5b617d4204a3" containerName="tempo" Oct 31 00:34:53.368769 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.368388 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="f26817af-cbdf-4baa-86b0-8b21f395779a" containerName="tempo" Oct 31 00:34:53.368769 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.368395 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="68837eb6-0f8f-4215-8771-1d661393310c" containerName="tempo" Oct 31 00:34:53.368769 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.368408 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="68837eb6-0f8f-4215-8771-1d661393310c" containerName="tempo" Oct 31 00:34:53.368769 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.368415 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb91015f-7854-47ed-9d1b-b81a71661356" containerName="otc-container" Oct 31 00:34:53.373196 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.373179 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:34:53.381072 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.379988 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:34:53.381072 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.380170 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-w5bd4\"" Oct 31 00:34:53.381072 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.380225 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-54f83bab\"" Oct 31 00:34:53.381072 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.380305 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:34:53.381072 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.380519 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:34:53.399924 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.399903 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/56a0d1a8-82bd-4e6d-b409-a7736d6adece-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"56a0d1a8-82bd-4e6d-b409-a7736d6adece\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:34:53.400034 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.399973 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/56a0d1a8-82bd-4e6d-b409-a7736d6adece-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"56a0d1a8-82bd-4e6d-b409-a7736d6adece\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:34:53.400075 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.400034 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj9ms\" (UniqueName: \"kubernetes.io/projected/56a0d1a8-82bd-4e6d-b409-a7736d6adece-kube-api-access-fj9ms\") pod \"data-science-collector-collector-1\" (UID: \"56a0d1a8-82bd-4e6d-b409-a7736d6adece\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:34:53.400127 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.400108 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/56a0d1a8-82bd-4e6d-b409-a7736d6adece-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"56a0d1a8-82bd-4e6d-b409-a7736d6adece\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:34:53.481553 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.481527 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68837eb6-0f8f-4215-8771-1d661393310c" path="/var/lib/kubelet/pods/68837eb6-0f8f-4215-8771-1d661393310c/volumes" Oct 31 00:34:53.481902 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.481889 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb91015f-7854-47ed-9d1b-b81a71661356" path="/var/lib/kubelet/pods/cb91015f-7854-47ed-9d1b-b81a71661356/volumes" Oct 31 00:34:53.500909 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.500891 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/56a0d1a8-82bd-4e6d-b409-a7736d6adece-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"56a0d1a8-82bd-4e6d-b409-a7736d6adece\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:34:53.500986 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.500950 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/56a0d1a8-82bd-4e6d-b409-a7736d6adece-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"56a0d1a8-82bd-4e6d-b409-a7736d6adece\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:34:53.501026 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.500998 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj9ms\" (UniqueName: \"kubernetes.io/projected/56a0d1a8-82bd-4e6d-b409-a7736d6adece-kube-api-access-fj9ms\") pod \"data-science-collector-collector-1\" (UID: \"56a0d1a8-82bd-4e6d-b409-a7736d6adece\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:34:53.501063 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.501054 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/56a0d1a8-82bd-4e6d-b409-a7736d6adece-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"56a0d1a8-82bd-4e6d-b409-a7736d6adece\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:34:53.501558 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.501538 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/56a0d1a8-82bd-4e6d-b409-a7736d6adece-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"56a0d1a8-82bd-4e6d-b409-a7736d6adece\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:34:53.503439 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.503417 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/56a0d1a8-82bd-4e6d-b409-a7736d6adece-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"56a0d1a8-82bd-4e6d-b409-a7736d6adece\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:34:53.503520 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.503458 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/56a0d1a8-82bd-4e6d-b409-a7736d6adece-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"56a0d1a8-82bd-4e6d-b409-a7736d6adece\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:34:53.513135 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.513107 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj9ms\" (UniqueName: \"kubernetes.io/projected/56a0d1a8-82bd-4e6d-b409-a7736d6adece-kube-api-access-fj9ms\") pod \"data-science-collector-collector-1\" (UID: \"56a0d1a8-82bd-4e6d-b409-a7736d6adece\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:34:53.688382 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.688360 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:34:53.962201 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.962150 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:34:53.964350 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:34:53.964324 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a0d1a8_82bd_4e6d_b409_a7736d6adece.slice/crio-c319bab65827f237e26bd3b547fa7f25febc51889dd50aed5cff0d4b191238a2 WatchSource:0}: Error finding container c319bab65827f237e26bd3b547fa7f25febc51889dd50aed5cff0d4b191238a2: Status 404 returned error can't find the container with id c319bab65827f237e26bd3b547fa7f25febc51889dd50aed5cff0d4b191238a2 Oct 31 00:34:53.966451 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.966420 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:34:53.966508 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:53.966480 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:34:54.280564 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:54.280478 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"56a0d1a8-82bd-4e6d-b409-a7736d6adece","Type":"ContainerStarted","Data":"6c0aebf6194b5971ae1d2c431795b18fefe0345234d591c1d46d5c0ef4f240d1"} Oct 31 00:34:54.280564 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:54.280531 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"56a0d1a8-82bd-4e6d-b409-a7736d6adece","Type":"ContainerStarted","Data":"c319bab65827f237e26bd3b547fa7f25febc51889dd50aed5cff0d4b191238a2"} Oct 31 00:34:54.302308 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:34:54.302239 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.302225129 podStartE2EDuration="1.302225129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:54.302015179 +0000 UTC m=+2130.938833898" watchObservedRunningTime="2026-10-31 00:34:54.302225129 +0000 UTC m=+2130.939043841" Oct 31 00:35:20.900042 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:35:20.899996 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:35:23.917513 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:23.917486 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-q62ss"] Oct 31 00:35:23.917869 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:23.917703 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-q62ss" podUID="27ca4a05-d89b-40c7-a71c-678ffa630bda" containerName="korrel8r" containerID="cri-o://27ea08e42682a00433f120251b124b92d8de58ca3735d8aa99dd5eacb1a77bc5" gracePeriod=30 Oct 31 00:35:24.038690 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.038661 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:35:24.039286 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.038923 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="56a0d1a8-82bd-4e6d-b409-a7736d6adece" containerName="otc-container" containerID="cri-o://6c0aebf6194b5971ae1d2c431795b18fefe0345234d591c1d46d5c0ef4f240d1" gracePeriod=30 Oct 31 00:35:24.206300 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.206274 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-q62ss" Oct 31 00:35:24.330061 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.330016 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27ca4a05-d89b-40c7-a71c-678ffa630bda-tls\" (UniqueName: \"kubernetes.io/secret/27ca4a05-d89b-40c7-a71c-678ffa630bda-tls\") pod \"27ca4a05-d89b-40c7-a71c-678ffa630bda\" (UID: \"27ca4a05-d89b-40c7-a71c-678ffa630bda\") " Oct 31 00:35:24.330703 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.330172 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27ca4a05-d89b-40c7-a71c-678ffa630bda-kube-api-access-t56f7\" (UniqueName: \"kubernetes.io/projected/27ca4a05-d89b-40c7-a71c-678ffa630bda-kube-api-access-t56f7\") pod \"27ca4a05-d89b-40c7-a71c-678ffa630bda\" (UID: \"27ca4a05-d89b-40c7-a71c-678ffa630bda\") " Oct 31 00:35:24.330703 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.330220 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27ca4a05-d89b-40c7-a71c-678ffa630bda-config\" (UniqueName: \"kubernetes.io/configmap/27ca4a05-d89b-40c7-a71c-678ffa630bda-config\") pod \"27ca4a05-d89b-40c7-a71c-678ffa630bda\" (UID: \"27ca4a05-d89b-40c7-a71c-678ffa630bda\") " Oct 31 00:35:24.332040 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.331947 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27ca4a05-d89b-40c7-a71c-678ffa630bda-config" pod "27ca4a05-d89b-40c7-a71c-678ffa630bda" (UID: "27ca4a05-d89b-40c7-a71c-678ffa630bda"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:24.333422 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.333397 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ca4a05-d89b-40c7-a71c-678ffa630bda-kube-api-access-t56f7" pod "27ca4a05-d89b-40c7-a71c-678ffa630bda" (UID: "27ca4a05-d89b-40c7-a71c-678ffa630bda"). InnerVolumeSpecName "kube-api-access-t56f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:24.333528 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.333425 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27ca4a05-d89b-40c7-a71c-678ffa630bda-tls" pod "27ca4a05-d89b-40c7-a71c-678ffa630bda" (UID: "27ca4a05-d89b-40c7-a71c-678ffa630bda"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:35:24.352483 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.352456 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-8469665bdf-d4mt9"] Oct 31 00:35:24.352692 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.352668 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-8469665bdf-d4mt9" podUID="5080ff17-a9b9-4a43-b6a1-70cc8dedd32d" containerName="tempo" containerID="cri-o://2907c0819986add472541751eff6c22477872cbc162c0acb87bfd147be28ba33" gracePeriod=30 Oct 31 00:35:24.361603 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.361584 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:24.433416 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.433331 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56a0d1a8-82bd-4e6d-b409-a7736d6adece-kube-api-access-fj9ms\" (UniqueName: \"kubernetes.io/projected/56a0d1a8-82bd-4e6d-b409-a7736d6adece-kube-api-access-fj9ms\") pod \"56a0d1a8-82bd-4e6d-b409-a7736d6adece\" (UID: \"56a0d1a8-82bd-4e6d-b409-a7736d6adece\") " Oct 31 00:35:24.433416 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.433390 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56a0d1a8-82bd-4e6d-b409-a7736d6adece-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/56a0d1a8-82bd-4e6d-b409-a7736d6adece-otlp-tls-certs\") pod \"56a0d1a8-82bd-4e6d-b409-a7736d6adece\" (UID: \"56a0d1a8-82bd-4e6d-b409-a7736d6adece\") " Oct 31 00:35:24.433650 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.433449 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56a0d1a8-82bd-4e6d-b409-a7736d6adece-otc-internal\" (UniqueName: \"kubernetes.io/configmap/56a0d1a8-82bd-4e6d-b409-a7736d6adece-otc-internal\") pod \"56a0d1a8-82bd-4e6d-b409-a7736d6adece\" (UID: \"56a0d1a8-82bd-4e6d-b409-a7736d6adece\") " Oct 31 00:35:24.433650 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.433480 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56a0d1a8-82bd-4e6d-b409-a7736d6adece-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/56a0d1a8-82bd-4e6d-b409-a7736d6adece-monitoring-tls-certs\") pod \"56a0d1a8-82bd-4e6d-b409-a7736d6adece\" (UID: \"56a0d1a8-82bd-4e6d-b409-a7736d6adece\") " Oct 31 00:35:24.433846 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.433808 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56a0d1a8-82bd-4e6d-b409-a7736d6adece-otc-internal" pod "56a0d1a8-82bd-4e6d-b409-a7736d6adece" (UID: "56a0d1a8-82bd-4e6d-b409-a7736d6adece"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:24.434000 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.433831 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t56f7\" (UniqueName: \"kubernetes.io/projected/27ca4a05-d89b-40c7-a71c-678ffa630bda-kube-api-access-t56f7\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:24.434137 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.434012 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/27ca4a05-d89b-40c7-a71c-678ffa630bda-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:24.434137 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.434033 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/27ca4a05-d89b-40c7-a71c-678ffa630bda-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:24.436058 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.435989 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a0d1a8-82bd-4e6d-b409-a7736d6adece-kube-api-access-fj9ms" pod "56a0d1a8-82bd-4e6d-b409-a7736d6adece" (UID: "56a0d1a8-82bd-4e6d-b409-a7736d6adece"). InnerVolumeSpecName "kube-api-access-fj9ms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:24.436265 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.436244 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56a0d1a8-82bd-4e6d-b409-a7736d6adece-monitoring-tls-certs" pod "56a0d1a8-82bd-4e6d-b409-a7736d6adece" (UID: "56a0d1a8-82bd-4e6d-b409-a7736d6adece"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:35:24.436484 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.436450 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56a0d1a8-82bd-4e6d-b409-a7736d6adece-otlp-tls-certs" pod "56a0d1a8-82bd-4e6d-b409-a7736d6adece" (UID: "56a0d1a8-82bd-4e6d-b409-a7736d6adece"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:35:24.437400 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.437373 5274 generic.go:356] "Generic (PLEG): container finished" podID="56a0d1a8-82bd-4e6d-b409-a7736d6adece" containerID="6c0aebf6194b5971ae1d2c431795b18fefe0345234d591c1d46d5c0ef4f240d1" exitCode=0 Oct 31 00:35:24.437494 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.437424 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:24.437494 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.437447 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"56a0d1a8-82bd-4e6d-b409-a7736d6adece","Type":"ContainerDied","Data":"6c0aebf6194b5971ae1d2c431795b18fefe0345234d591c1d46d5c0ef4f240d1"} Oct 31 00:35:24.437609 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.437526 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"56a0d1a8-82bd-4e6d-b409-a7736d6adece","Type":"ContainerDied","Data":"c319bab65827f237e26bd3b547fa7f25febc51889dd50aed5cff0d4b191238a2"} Oct 31 00:35:24.437609 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.437558 5274 scope.go:122] "RemoveContainer" containerID="6c0aebf6194b5971ae1d2c431795b18fefe0345234d591c1d46d5c0ef4f240d1" Oct 31 00:35:24.438950 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.438932 5274 generic.go:356] "Generic (PLEG): container finished" podID="27ca4a05-d89b-40c7-a71c-678ffa630bda" containerID="27ea08e42682a00433f120251b124b92d8de58ca3735d8aa99dd5eacb1a77bc5" exitCode=0 Oct 31 00:35:24.438950 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.439011 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-q62ss" Oct 31 00:35:24.439142 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.439002 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-q62ss" event={"ID":"27ca4a05-d89b-40c7-a71c-678ffa630bda","Type":"ContainerDied","Data":"27ea08e42682a00433f120251b124b92d8de58ca3735d8aa99dd5eacb1a77bc5"} Oct 31 00:35:24.439142 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.439135 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-q62ss" event={"ID":"27ca4a05-d89b-40c7-a71c-678ffa630bda","Type":"ContainerDied","Data":"844d4a5b51c4dac740ccb2c650aa31c98be288c8e3ef7f0e492d961279ae9bae"} Oct 31 00:35:24.465975 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.465935 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:35:24.467090 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.467077 5274 scope.go:122] "RemoveContainer" containerID="6c0aebf6194b5971ae1d2c431795b18fefe0345234d591c1d46d5c0ef4f240d1" Oct 31 00:35:24.467413 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:35:24.467394 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c0aebf6194b5971ae1d2c431795b18fefe0345234d591c1d46d5c0ef4f240d1\": container with ID starting with 6c0aebf6194b5971ae1d2c431795b18fefe0345234d591c1d46d5c0ef4f240d1 not found: ID does not exist" containerID="6c0aebf6194b5971ae1d2c431795b18fefe0345234d591c1d46d5c0ef4f240d1" Oct 31 00:35:24.467500 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.467420 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c0aebf6194b5971ae1d2c431795b18fefe0345234d591c1d46d5c0ef4f240d1"} err="failed to get container status \"6c0aebf6194b5971ae1d2c431795b18fefe0345234d591c1d46d5c0ef4f240d1\": rpc error: code = NotFound desc = could not find container \"6c0aebf6194b5971ae1d2c431795b18fefe0345234d591c1d46d5c0ef4f240d1\": container with ID starting with 6c0aebf6194b5971ae1d2c431795b18fefe0345234d591c1d46d5c0ef4f240d1 not found: ID does not exist" Oct 31 00:35:24.467500 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.467435 5274 scope.go:122] "RemoveContainer" containerID="27ea08e42682a00433f120251b124b92d8de58ca3735d8aa99dd5eacb1a77bc5" Oct 31 00:35:24.473700 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.473678 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:35:24.476179 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.476159 5274 scope.go:122] "RemoveContainer" containerID="27ea08e42682a00433f120251b124b92d8de58ca3735d8aa99dd5eacb1a77bc5" Oct 31 00:35:24.476460 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:35:24.476440 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27ea08e42682a00433f120251b124b92d8de58ca3735d8aa99dd5eacb1a77bc5\": container with ID starting with 27ea08e42682a00433f120251b124b92d8de58ca3735d8aa99dd5eacb1a77bc5 not found: ID does not exist" containerID="27ea08e42682a00433f120251b124b92d8de58ca3735d8aa99dd5eacb1a77bc5" Oct 31 00:35:24.476533 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.476470 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27ea08e42682a00433f120251b124b92d8de58ca3735d8aa99dd5eacb1a77bc5"} err="failed to get container status \"27ea08e42682a00433f120251b124b92d8de58ca3735d8aa99dd5eacb1a77bc5\": rpc error: code = NotFound desc = could not find container \"27ea08e42682a00433f120251b124b92d8de58ca3735d8aa99dd5eacb1a77bc5\": container with ID starting with 27ea08e42682a00433f120251b124b92d8de58ca3735d8aa99dd5eacb1a77bc5 not found: ID does not exist" Oct 31 00:35:24.484100 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.484076 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-q62ss"] Oct 31 00:35:24.490295 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.490270 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-q62ss"] Oct 31 00:35:24.534446 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.534422 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj9ms\" (UniqueName: \"kubernetes.io/projected/56a0d1a8-82bd-4e6d-b409-a7736d6adece-kube-api-access-fj9ms\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:24.534560 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.534449 5274 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/56a0d1a8-82bd-4e6d-b409-a7736d6adece-otlp-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:24.534560 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.534466 5274 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/56a0d1a8-82bd-4e6d-b409-a7736d6adece-otc-internal\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:24.534560 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.534478 5274 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/56a0d1a8-82bd-4e6d-b409-a7736d6adece-monitoring-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:24.590302 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.590279 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-8469665bdf-d4mt9" Oct 31 00:35:24.635491 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.635467 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5080ff17-a9b9-4a43-b6a1-70cc8dedd32d-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5080ff17-a9b9-4a43-b6a1-70cc8dedd32d-tempo-data-science-tempostack-ca-bundle\") pod \"5080ff17-a9b9-4a43-b6a1-70cc8dedd32d\" (UID: \"5080ff17-a9b9-4a43-b6a1-70cc8dedd32d\") " Oct 31 00:35:24.635624 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.635508 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5080ff17-a9b9-4a43-b6a1-70cc8dedd32d-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/5080ff17-a9b9-4a43-b6a1-70cc8dedd32d-tempo-conf\") pod \"5080ff17-a9b9-4a43-b6a1-70cc8dedd32d\" (UID: \"5080ff17-a9b9-4a43-b6a1-70cc8dedd32d\") " Oct 31 00:35:24.635624 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.635532 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5080ff17-a9b9-4a43-b6a1-70cc8dedd32d-tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/5080ff17-a9b9-4a43-b6a1-70cc8dedd32d-tempo-data-science-tempostack-distributor-mtls\") pod \"5080ff17-a9b9-4a43-b6a1-70cc8dedd32d\" (UID: \"5080ff17-a9b9-4a43-b6a1-70cc8dedd32d\") " Oct 31 00:35:24.635624 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.635609 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5080ff17-a9b9-4a43-b6a1-70cc8dedd32d-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/5080ff17-a9b9-4a43-b6a1-70cc8dedd32d-tempo-tmp-storage\") pod \"5080ff17-a9b9-4a43-b6a1-70cc8dedd32d\" (UID: \"5080ff17-a9b9-4a43-b6a1-70cc8dedd32d\") " Oct 31 00:35:24.635797 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.635634 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5080ff17-a9b9-4a43-b6a1-70cc8dedd32d-kube-api-access-tcdfp\" (UniqueName: \"kubernetes.io/projected/5080ff17-a9b9-4a43-b6a1-70cc8dedd32d-kube-api-access-tcdfp\") pod \"5080ff17-a9b9-4a43-b6a1-70cc8dedd32d\" (UID: \"5080ff17-a9b9-4a43-b6a1-70cc8dedd32d\") " Oct 31 00:35:24.635906 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.635885 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5080ff17-a9b9-4a43-b6a1-70cc8dedd32d-tempo-tmp-storage" pod "5080ff17-a9b9-4a43-b6a1-70cc8dedd32d" (UID: "5080ff17-a9b9-4a43-b6a1-70cc8dedd32d"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:35:24.635987 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.635920 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5080ff17-a9b9-4a43-b6a1-70cc8dedd32d-tempo-data-science-tempostack-ca-bundle" pod "5080ff17-a9b9-4a43-b6a1-70cc8dedd32d" (UID: "5080ff17-a9b9-4a43-b6a1-70cc8dedd32d"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:24.636036 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.635988 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5080ff17-a9b9-4a43-b6a1-70cc8dedd32d-tempo-conf" pod "5080ff17-a9b9-4a43-b6a1-70cc8dedd32d" (UID: "5080ff17-a9b9-4a43-b6a1-70cc8dedd32d"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:24.637615 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.637598 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5080ff17-a9b9-4a43-b6a1-70cc8dedd32d-tempo-data-science-tempostack-distributor-mtls" pod "5080ff17-a9b9-4a43-b6a1-70cc8dedd32d" (UID: "5080ff17-a9b9-4a43-b6a1-70cc8dedd32d"). InnerVolumeSpecName "tempo-data-science-tempostack-distributor-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:35:24.637668 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.637630 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5080ff17-a9b9-4a43-b6a1-70cc8dedd32d-kube-api-access-tcdfp" pod "5080ff17-a9b9-4a43-b6a1-70cc8dedd32d" (UID: "5080ff17-a9b9-4a43-b6a1-70cc8dedd32d"). InnerVolumeSpecName "kube-api-access-tcdfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:24.737045 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.736987 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5080ff17-a9b9-4a43-b6a1-70cc8dedd32d-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:24.737045 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.737010 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/5080ff17-a9b9-4a43-b6a1-70cc8dedd32d-tempo-conf\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:24.737045 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.737020 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/5080ff17-a9b9-4a43-b6a1-70cc8dedd32d-tempo-data-science-tempostack-distributor-mtls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:24.737045 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.737032 5274 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/5080ff17-a9b9-4a43-b6a1-70cc8dedd32d-tempo-tmp-storage\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:24.737045 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:24.737041 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcdfp\" (UniqueName: \"kubernetes.io/projected/5080ff17-a9b9-4a43-b6a1-70cc8dedd32d-kube-api-access-tcdfp\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:25.445940 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:25.445903 5274 generic.go:356] "Generic (PLEG): container finished" podID="5080ff17-a9b9-4a43-b6a1-70cc8dedd32d" containerID="2907c0819986add472541751eff6c22477872cbc162c0acb87bfd147be28ba33" exitCode=0 Oct 31 00:35:25.446378 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:25.445972 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-8469665bdf-d4mt9" Oct 31 00:35:25.446378 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:25.445984 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-8469665bdf-d4mt9" event={"ID":"5080ff17-a9b9-4a43-b6a1-70cc8dedd32d","Type":"ContainerDied","Data":"2907c0819986add472541751eff6c22477872cbc162c0acb87bfd147be28ba33"} Oct 31 00:35:25.446378 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:25.446037 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-8469665bdf-d4mt9" event={"ID":"5080ff17-a9b9-4a43-b6a1-70cc8dedd32d","Type":"ContainerDied","Data":"a6f02bbcb25ece18f9784722b2566bf59792eb12743518f0634cce6e34a61d3d"} Oct 31 00:35:25.446378 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:25.446068 5274 scope.go:122] "RemoveContainer" containerID="2907c0819986add472541751eff6c22477872cbc162c0acb87bfd147be28ba33" Oct 31 00:35:25.455108 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:25.455091 5274 scope.go:122] "RemoveContainer" containerID="2907c0819986add472541751eff6c22477872cbc162c0acb87bfd147be28ba33" Oct 31 00:35:25.455372 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:35:25.455353 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2907c0819986add472541751eff6c22477872cbc162c0acb87bfd147be28ba33\": container with ID starting with 2907c0819986add472541751eff6c22477872cbc162c0acb87bfd147be28ba33 not found: ID does not exist" containerID="2907c0819986add472541751eff6c22477872cbc162c0acb87bfd147be28ba33" Oct 31 00:35:25.455442 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:25.455385 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2907c0819986add472541751eff6c22477872cbc162c0acb87bfd147be28ba33"} err="failed to get container status \"2907c0819986add472541751eff6c22477872cbc162c0acb87bfd147be28ba33\": rpc error: code = NotFound desc = could not find container \"2907c0819986add472541751eff6c22477872cbc162c0acb87bfd147be28ba33\": container with ID starting with 2907c0819986add472541751eff6c22477872cbc162c0acb87bfd147be28ba33 not found: ID does not exist" Oct 31 00:35:25.467942 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:25.467919 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-8469665bdf-d4mt9"] Oct 31 00:35:25.472826 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:25.472805 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-8469665bdf-d4mt9"] Oct 31 00:35:25.480704 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:25.480683 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27ca4a05-d89b-40c7-a71c-678ffa630bda" path="/var/lib/kubelet/pods/27ca4a05-d89b-40c7-a71c-678ffa630bda/volumes" Oct 31 00:35:25.481160 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:25.481143 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5080ff17-a9b9-4a43-b6a1-70cc8dedd32d" path="/var/lib/kubelet/pods/5080ff17-a9b9-4a43-b6a1-70cc8dedd32d/volumes" Oct 31 00:35:25.481534 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:25.481517 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56a0d1a8-82bd-4e6d-b409-a7736d6adece" path="/var/lib/kubelet/pods/56a0d1a8-82bd-4e6d-b409-a7736d6adece/volumes" Oct 31 00:35:28.823191 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:28.823157 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-grj2s"] Oct 31 00:35:28.823592 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:28.823503 5274 cpu_manager.go:414] "containerMap: removing container" podUID="56a0d1a8-82bd-4e6d-b409-a7736d6adece" containerName="otc-container" Oct 31 00:35:28.823592 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:28.823513 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56a0d1a8-82bd-4e6d-b409-a7736d6adece" containerName="otc-container" Oct 31 00:35:28.823592 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:28.823526 5274 cpu_manager.go:414] "containerMap: removing container" podUID="27ca4a05-d89b-40c7-a71c-678ffa630bda" containerName="korrel8r" Oct 31 00:35:28.823592 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:28.823531 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27ca4a05-d89b-40c7-a71c-678ffa630bda" containerName="korrel8r" Oct 31 00:35:28.823592 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:28.823545 5274 cpu_manager.go:414] "containerMap: removing container" podUID="5080ff17-a9b9-4a43-b6a1-70cc8dedd32d" containerName="tempo" Oct 31 00:35:28.823592 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:28.823551 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5080ff17-a9b9-4a43-b6a1-70cc8dedd32d" containerName="tempo" Oct 31 00:35:28.823794 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:28.823597 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="56a0d1a8-82bd-4e6d-b409-a7736d6adece" containerName="otc-container" Oct 31 00:35:28.823794 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:28.823604 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="27ca4a05-d89b-40c7-a71c-678ffa630bda" containerName="korrel8r" Oct 31 00:35:28.823794 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:28.823611 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="5080ff17-a9b9-4a43-b6a1-70cc8dedd32d" containerName="tempo" Oct 31 00:35:28.828120 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:28.828100 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-grj2s" Oct 31 00:35:28.835410 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:28.835385 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-grj2s"] Oct 31 00:35:28.836254 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:28.836232 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-wkk4q\"" Oct 31 00:35:28.836385 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:28.836330 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:35:28.836480 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:28.836463 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 31 00:35:28.836532 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:28.836464 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:35:28.838027 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:28.838005 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 31 00:35:28.969045 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:28.969012 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fxj2\" (UniqueName: \"kubernetes.io/projected/9ad13ab8-0a74-409a-ad75-2f8537ab06d0-kube-api-access-4fxj2\") pod \"korrel8r-6644769df5-grj2s\" (UID: \"9ad13ab8-0a74-409a-ad75-2f8537ab06d0\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-grj2s" Oct 31 00:35:28.969223 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:28.969068 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9ad13ab8-0a74-409a-ad75-2f8537ab06d0-config\") pod \"korrel8r-6644769df5-grj2s\" (UID: \"9ad13ab8-0a74-409a-ad75-2f8537ab06d0\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-grj2s" Oct 31 00:35:28.969223 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:28.969166 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/9ad13ab8-0a74-409a-ad75-2f8537ab06d0-tls\") pod \"korrel8r-6644769df5-grj2s\" (UID: \"9ad13ab8-0a74-409a-ad75-2f8537ab06d0\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-grj2s" Oct 31 00:35:29.070207 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:29.070167 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9ad13ab8-0a74-409a-ad75-2f8537ab06d0-config\") pod \"korrel8r-6644769df5-grj2s\" (UID: \"9ad13ab8-0a74-409a-ad75-2f8537ab06d0\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-grj2s" Oct 31 00:35:29.070401 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:29.070251 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/9ad13ab8-0a74-409a-ad75-2f8537ab06d0-tls\") pod \"korrel8r-6644769df5-grj2s\" (UID: \"9ad13ab8-0a74-409a-ad75-2f8537ab06d0\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-grj2s" Oct 31 00:35:29.070401 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:29.070299 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fxj2\" (UniqueName: \"kubernetes.io/projected/9ad13ab8-0a74-409a-ad75-2f8537ab06d0-kube-api-access-4fxj2\") pod \"korrel8r-6644769df5-grj2s\" (UID: \"9ad13ab8-0a74-409a-ad75-2f8537ab06d0\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-grj2s" Oct 31 00:35:29.071690 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:29.071659 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9ad13ab8-0a74-409a-ad75-2f8537ab06d0-config\") pod \"korrel8r-6644769df5-grj2s\" (UID: \"9ad13ab8-0a74-409a-ad75-2f8537ab06d0\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-grj2s" Oct 31 00:35:29.074240 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:29.074183 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/9ad13ab8-0a74-409a-ad75-2f8537ab06d0-tls\") pod \"korrel8r-6644769df5-grj2s\" (UID: \"9ad13ab8-0a74-409a-ad75-2f8537ab06d0\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-grj2s" Oct 31 00:35:29.095562 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:29.095538 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fxj2\" (UniqueName: \"kubernetes.io/projected/9ad13ab8-0a74-409a-ad75-2f8537ab06d0-kube-api-access-4fxj2\") pod \"korrel8r-6644769df5-grj2s\" (UID: \"9ad13ab8-0a74-409a-ad75-2f8537ab06d0\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-grj2s" Oct 31 00:35:29.141536 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:29.141510 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-grj2s" Oct 31 00:35:29.427009 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:29.422785 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-grj2s"] Oct 31 00:35:29.470565 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:29.470534 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-grj2s" event={"ID":"9ad13ab8-0a74-409a-ad75-2f8537ab06d0","Type":"ContainerStarted","Data":"7e166568c505696187761ef81e700323c1faebe225a1d2ce1c6ea54c85782f6c"} Oct 31 00:35:30.477167 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:30.477129 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-grj2s" event={"ID":"9ad13ab8-0a74-409a-ad75-2f8537ab06d0","Type":"ContainerStarted","Data":"4dd0c8d5e89d01f9d7438feb3f7ca4c7de38837d5cbc5a7ba7682992342e14b0"} Oct 31 00:35:30.477567 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:30.477280 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-grj2s" Oct 31 00:35:30.497029 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:30.496991 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-6644769df5-grj2s" podStartSLOduration=2.496979715 podStartE2EDuration="2.496979715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:30.495764608 +0000 UTC m=+2167.132583325" watchObservedRunningTime="2026-10-31 00:35:30.496979715 +0000 UTC m=+2167.133798430" Oct 31 00:35:41.484465 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:41.484397 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-grj2s" Oct 31 00:35:44.767220 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:44.767187 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-grj2s"] Oct 31 00:35:44.767624 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:44.767370 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-6644769df5-grj2s" podUID="9ad13ab8-0a74-409a-ad75-2f8537ab06d0" containerName="korrel8r" containerID="cri-o://4dd0c8d5e89d01f9d7438feb3f7ca4c7de38837d5cbc5a7ba7682992342e14b0" gracePeriod=30 Oct 31 00:35:45.043780 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:45.043755 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-grj2s" Oct 31 00:35:45.205830 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:45.205798 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ad13ab8-0a74-409a-ad75-2f8537ab06d0-tls\" (UniqueName: \"kubernetes.io/secret/9ad13ab8-0a74-409a-ad75-2f8537ab06d0-tls\") pod \"9ad13ab8-0a74-409a-ad75-2f8537ab06d0\" (UID: \"9ad13ab8-0a74-409a-ad75-2f8537ab06d0\") " Oct 31 00:35:45.206020 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:45.205847 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ad13ab8-0a74-409a-ad75-2f8537ab06d0-config\" (UniqueName: \"kubernetes.io/configmap/9ad13ab8-0a74-409a-ad75-2f8537ab06d0-config\") pod \"9ad13ab8-0a74-409a-ad75-2f8537ab06d0\" (UID: \"9ad13ab8-0a74-409a-ad75-2f8537ab06d0\") " Oct 31 00:35:45.206020 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:45.205912 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ad13ab8-0a74-409a-ad75-2f8537ab06d0-kube-api-access-4fxj2\" (UniqueName: \"kubernetes.io/projected/9ad13ab8-0a74-409a-ad75-2f8537ab06d0-kube-api-access-4fxj2\") pod \"9ad13ab8-0a74-409a-ad75-2f8537ab06d0\" (UID: \"9ad13ab8-0a74-409a-ad75-2f8537ab06d0\") " Oct 31 00:35:45.206376 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:45.206332 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ad13ab8-0a74-409a-ad75-2f8537ab06d0-config" pod "9ad13ab8-0a74-409a-ad75-2f8537ab06d0" (UID: "9ad13ab8-0a74-409a-ad75-2f8537ab06d0"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:45.207874 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:45.207853 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ad13ab8-0a74-409a-ad75-2f8537ab06d0-tls" pod "9ad13ab8-0a74-409a-ad75-2f8537ab06d0" (UID: "9ad13ab8-0a74-409a-ad75-2f8537ab06d0"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:35:45.208051 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:45.208037 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ad13ab8-0a74-409a-ad75-2f8537ab06d0-kube-api-access-4fxj2" pod "9ad13ab8-0a74-409a-ad75-2f8537ab06d0" (UID: "9ad13ab8-0a74-409a-ad75-2f8537ab06d0"). InnerVolumeSpecName "kube-api-access-4fxj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:45.306692 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:45.306633 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fxj2\" (UniqueName: \"kubernetes.io/projected/9ad13ab8-0a74-409a-ad75-2f8537ab06d0-kube-api-access-4fxj2\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:45.306692 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:45.306658 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/9ad13ab8-0a74-409a-ad75-2f8537ab06d0-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:45.306692 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:45.306669 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9ad13ab8-0a74-409a-ad75-2f8537ab06d0-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:45.546065 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:45.546037 5274 generic.go:356] "Generic (PLEG): container finished" podID="9ad13ab8-0a74-409a-ad75-2f8537ab06d0" containerID="4dd0c8d5e89d01f9d7438feb3f7ca4c7de38837d5cbc5a7ba7682992342e14b0" exitCode=0 Oct 31 00:35:45.546230 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:45.546107 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-grj2s" Oct 31 00:35:45.546230 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:45.546124 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-grj2s" event={"ID":"9ad13ab8-0a74-409a-ad75-2f8537ab06d0","Type":"ContainerDied","Data":"4dd0c8d5e89d01f9d7438feb3f7ca4c7de38837d5cbc5a7ba7682992342e14b0"} Oct 31 00:35:45.546230 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:45.546172 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-grj2s" event={"ID":"9ad13ab8-0a74-409a-ad75-2f8537ab06d0","Type":"ContainerDied","Data":"7e166568c505696187761ef81e700323c1faebe225a1d2ce1c6ea54c85782f6c"} Oct 31 00:35:45.546230 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:45.546198 5274 scope.go:122] "RemoveContainer" containerID="4dd0c8d5e89d01f9d7438feb3f7ca4c7de38837d5cbc5a7ba7682992342e14b0" Oct 31 00:35:45.554708 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:45.554675 5274 scope.go:122] "RemoveContainer" containerID="4dd0c8d5e89d01f9d7438feb3f7ca4c7de38837d5cbc5a7ba7682992342e14b0" Oct 31 00:35:45.555004 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:35:45.554982 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dd0c8d5e89d01f9d7438feb3f7ca4c7de38837d5cbc5a7ba7682992342e14b0\": container with ID starting with 4dd0c8d5e89d01f9d7438feb3f7ca4c7de38837d5cbc5a7ba7682992342e14b0 not found: ID does not exist" containerID="4dd0c8d5e89d01f9d7438feb3f7ca4c7de38837d5cbc5a7ba7682992342e14b0" Oct 31 00:35:45.555056 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:45.555013 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dd0c8d5e89d01f9d7438feb3f7ca4c7de38837d5cbc5a7ba7682992342e14b0"} err="failed to get container status \"4dd0c8d5e89d01f9d7438feb3f7ca4c7de38837d5cbc5a7ba7682992342e14b0\": rpc error: code = NotFound desc = could not find container \"4dd0c8d5e89d01f9d7438feb3f7ca4c7de38837d5cbc5a7ba7682992342e14b0\": container with ID starting with 4dd0c8d5e89d01f9d7438feb3f7ca4c7de38837d5cbc5a7ba7682992342e14b0 not found: ID does not exist" Oct 31 00:35:45.568054 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:45.568008 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-grj2s"] Oct 31 00:35:45.572916 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:45.572895 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-grj2s"] Oct 31 00:35:47.480729 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:47.480697 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ad13ab8-0a74-409a-ad75-2f8537ab06d0" path="/var/lib/kubelet/pods/9ad13ab8-0a74-409a-ad75-2f8537ab06d0/volumes" Oct 31 00:35:50.898112 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:35:50.898076 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:35:53.408835 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.408804 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k28gg"] Oct 31 00:35:53.409254 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.409239 5274 cpu_manager.go:414] "containerMap: removing container" podUID="9ad13ab8-0a74-409a-ad75-2f8537ab06d0" containerName="korrel8r" Oct 31 00:35:53.409302 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.409255 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ad13ab8-0a74-409a-ad75-2f8537ab06d0" containerName="korrel8r" Oct 31 00:35:53.409336 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.409318 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ad13ab8-0a74-409a-ad75-2f8537ab06d0" containerName="korrel8r" Oct 31 00:35:53.412269 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.412252 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k28gg" Oct 31 00:35:53.418025 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.418006 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:35:53.418025 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.418017 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:35:53.418174 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.418067 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:35:53.419220 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.419192 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\"" Oct 31 00:35:53.419320 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.419306 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-mpngg\"" Oct 31 00:35:53.421713 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.421695 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 31 00:35:53.421824 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.421810 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\"" Oct 31 00:35:53.425493 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.425474 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k28gg"] Oct 31 00:35:53.482928 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.482904 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-wdnls"] Oct 31 00:35:53.486111 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.486094 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wdnls" Oct 31 00:35:53.494940 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.494917 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-wdnls"] Oct 31 00:35:53.518237 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.518214 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 31 00:35:53.519889 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.519869 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 31 00:35:53.520248 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.520227 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-clgjc\"" Oct 31 00:35:53.572686 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.572662 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nfgh\" (UniqueName: \"kubernetes.io/projected/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-kube-api-access-7nfgh\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-k28gg\" (UID: \"3dc6465e-bbd3-4e25-b4c6-d28231ef86d6\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k28gg" Oct 31 00:35:53.572781 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.572705 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-k28gg\" (UID: \"3dc6465e-bbd3-4e25-b4c6-d28231ef86d6\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k28gg" Oct 31 00:35:53.572781 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.572728 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-k28gg\" (UID: \"3dc6465e-bbd3-4e25-b4c6-d28231ef86d6\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k28gg" Oct 31 00:35:53.572870 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.572827 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-k28gg\" (UID: \"3dc6465e-bbd3-4e25-b4c6-d28231ef86d6\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k28gg" Oct 31 00:35:53.572870 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.572853 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-k28gg\" (UID: \"3dc6465e-bbd3-4e25-b4c6-d28231ef86d6\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k28gg" Oct 31 00:35:53.673527 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.673454 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk2sw\" (UniqueName: \"kubernetes.io/projected/72eedfa0-5a3d-4047-98ea-c5e9f5582552-kube-api-access-nk2sw\") pod \"korrel8r-5cbcc89b8f-wdnls\" (UID: \"72eedfa0-5a3d-4047-98ea-c5e9f5582552\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wdnls" Oct 31 00:35:53.673527 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.673499 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nfgh\" (UniqueName: \"kubernetes.io/projected/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-kube-api-access-7nfgh\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-k28gg\" (UID: \"3dc6465e-bbd3-4e25-b4c6-d28231ef86d6\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k28gg" Oct 31 00:35:53.673738 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.673531 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-k28gg\" (UID: \"3dc6465e-bbd3-4e25-b4c6-d28231ef86d6\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k28gg" Oct 31 00:35:53.673738 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.673585 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/72eedfa0-5a3d-4047-98ea-c5e9f5582552-tls\") pod \"korrel8r-5cbcc89b8f-wdnls\" (UID: \"72eedfa0-5a3d-4047-98ea-c5e9f5582552\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wdnls" Oct 31 00:35:53.673738 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.673625 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-k28gg\" (UID: \"3dc6465e-bbd3-4e25-b4c6-d28231ef86d6\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k28gg" Oct 31 00:35:53.673738 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.673656 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-k28gg\" (UID: \"3dc6465e-bbd3-4e25-b4c6-d28231ef86d6\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k28gg" Oct 31 00:35:53.673738 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.673674 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-k28gg\" (UID: \"3dc6465e-bbd3-4e25-b4c6-d28231ef86d6\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k28gg" Oct 31 00:35:53.673988 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.673737 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/72eedfa0-5a3d-4047-98ea-c5e9f5582552-config\") pod \"korrel8r-5cbcc89b8f-wdnls\" (UID: \"72eedfa0-5a3d-4047-98ea-c5e9f5582552\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wdnls" Oct 31 00:35:53.673988 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:35:53.673742 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found Oct 31 00:35:53.673988 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:35:53.673837 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-prometheus-client-cert podName:3dc6465e-bbd3-4e25-b4c6-d28231ef86d6 nodeName:}" failed. No retries permitted until 2026-10-31 00:35:54.173821807 +0000 UTC m=+2190.810640500 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-k28gg" (UID: "3dc6465e-bbd3-4e25-b4c6-d28231ef86d6") : secret "prometheus-operated-tls" not found Oct 31 00:35:53.674378 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.674359 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-k28gg\" (UID: \"3dc6465e-bbd3-4e25-b4c6-d28231ef86d6\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k28gg" Oct 31 00:35:53.676043 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.676022 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-k28gg\" (UID: \"3dc6465e-bbd3-4e25-b4c6-d28231ef86d6\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k28gg" Oct 31 00:35:53.676125 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.676022 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-k28gg\" (UID: \"3dc6465e-bbd3-4e25-b4c6-d28231ef86d6\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k28gg" Oct 31 00:35:53.685889 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.685865 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nfgh\" (UniqueName: \"kubernetes.io/projected/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-kube-api-access-7nfgh\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-k28gg\" (UID: \"3dc6465e-bbd3-4e25-b4c6-d28231ef86d6\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k28gg" Oct 31 00:35:53.775060 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.775031 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/72eedfa0-5a3d-4047-98ea-c5e9f5582552-tls\") pod \"korrel8r-5cbcc89b8f-wdnls\" (UID: \"72eedfa0-5a3d-4047-98ea-c5e9f5582552\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wdnls" Oct 31 00:35:53.775180 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.775117 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/72eedfa0-5a3d-4047-98ea-c5e9f5582552-config\") pod \"korrel8r-5cbcc89b8f-wdnls\" (UID: \"72eedfa0-5a3d-4047-98ea-c5e9f5582552\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wdnls" Oct 31 00:35:53.775180 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.775150 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk2sw\" (UniqueName: \"kubernetes.io/projected/72eedfa0-5a3d-4047-98ea-c5e9f5582552-kube-api-access-nk2sw\") pod \"korrel8r-5cbcc89b8f-wdnls\" (UID: \"72eedfa0-5a3d-4047-98ea-c5e9f5582552\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wdnls" Oct 31 00:35:53.775870 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.775845 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/72eedfa0-5a3d-4047-98ea-c5e9f5582552-config\") pod \"korrel8r-5cbcc89b8f-wdnls\" (UID: \"72eedfa0-5a3d-4047-98ea-c5e9f5582552\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wdnls" Oct 31 00:35:53.777180 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.777162 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/72eedfa0-5a3d-4047-98ea-c5e9f5582552-tls\") pod \"korrel8r-5cbcc89b8f-wdnls\" (UID: \"72eedfa0-5a3d-4047-98ea-c5e9f5582552\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wdnls" Oct 31 00:35:53.787145 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.787120 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk2sw\" (UniqueName: \"kubernetes.io/projected/72eedfa0-5a3d-4047-98ea-c5e9f5582552-kube-api-access-nk2sw\") pod \"korrel8r-5cbcc89b8f-wdnls\" (UID: \"72eedfa0-5a3d-4047-98ea-c5e9f5582552\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wdnls" Oct 31 00:35:53.800006 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.799982 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wdnls" Oct 31 00:35:53.933259 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.933189 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:35:53.936970 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.936941 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:53.945312 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.945221 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:35:53.945312 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.945247 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:35:53.945312 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.945285 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:35:53.945312 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.945285 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:35:53.945615 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.945426 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:35:53.949817 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:53.949791 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:35:54.079191 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:54.079163 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a2746a63-fc36-4b4f-a17a-a31e50053bfd-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"a2746a63-fc36-4b4f-a17a-a31e50053bfd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:54.079191 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:54.079194 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a2746a63-fc36-4b4f-a17a-a31e50053bfd-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"a2746a63-fc36-4b4f-a17a-a31e50053bfd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:54.079424 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:54.079216 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77c7t\" (UniqueName: \"kubernetes.io/projected/a2746a63-fc36-4b4f-a17a-a31e50053bfd-kube-api-access-77c7t\") pod \"data-science-collector-collector-0\" (UID: \"a2746a63-fc36-4b4f-a17a-a31e50053bfd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:54.079424 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:54.079239 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a2746a63-fc36-4b4f-a17a-a31e50053bfd-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a2746a63-fc36-4b4f-a17a-a31e50053bfd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:54.079424 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:54.079309 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2746a63-fc36-4b4f-a17a-a31e50053bfd-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a2746a63-fc36-4b4f-a17a-a31e50053bfd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:54.079520 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:54.079447 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a2746a63-fc36-4b4f-a17a-a31e50053bfd-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a2746a63-fc36-4b4f-a17a-a31e50053bfd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:54.088080 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:54.087914 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-wdnls"] Oct 31 00:35:54.090975 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:35:54.090938 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72eedfa0_5a3d_4047_98ea_c5e9f5582552.slice/crio-5bac39bff688ae2382dcd7423982cd1ffa527758125be94cc4ff7e2250a05855 WatchSource:0}: Error finding container 5bac39bff688ae2382dcd7423982cd1ffa527758125be94cc4ff7e2250a05855: Status 404 returned error can't find the container with id 5bac39bff688ae2382dcd7423982cd1ffa527758125be94cc4ff7e2250a05855 Oct 31 00:35:54.180320 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:54.180292 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77c7t\" (UniqueName: \"kubernetes.io/projected/a2746a63-fc36-4b4f-a17a-a31e50053bfd-kube-api-access-77c7t\") pod \"data-science-collector-collector-0\" (UID: \"a2746a63-fc36-4b4f-a17a-a31e50053bfd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:54.180445 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:54.180335 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a2746a63-fc36-4b4f-a17a-a31e50053bfd-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a2746a63-fc36-4b4f-a17a-a31e50053bfd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:54.180445 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:54.180355 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2746a63-fc36-4b4f-a17a-a31e50053bfd-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a2746a63-fc36-4b4f-a17a-a31e50053bfd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:54.180560 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:35:54.180444 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found Oct 31 00:35:54.180560 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:35:54.180501 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found Oct 31 00:35:54.180560 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:54.180440 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-k28gg\" (UID: \"3dc6465e-bbd3-4e25-b4c6-d28231ef86d6\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k28gg" Oct 31 00:35:54.180560 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:35:54.180533 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2746a63-fc36-4b4f-a17a-a31e50053bfd-otlp-tls-certs podName:a2746a63-fc36-4b4f-a17a-a31e50053bfd nodeName:}" failed. No retries permitted until 2026-10-31 00:35:54.680503189 +0000 UTC m=+2191.317321897 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/a2746a63-fc36-4b4f-a17a-a31e50053bfd-otlp-tls-certs") pod "data-science-collector-collector-0" (UID: "a2746a63-fc36-4b4f-a17a-a31e50053bfd") : secret "data-science-collector-collector-headless-tls" not found Oct 31 00:35:54.180720 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:35:54.180631 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-prometheus-client-cert podName:3dc6465e-bbd3-4e25-b4c6-d28231ef86d6 nodeName:}" failed. No retries permitted until 2026-10-31 00:35:55.180604546 +0000 UTC m=+2191.817423253 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-k28gg" (UID: "3dc6465e-bbd3-4e25-b4c6-d28231ef86d6") : secret "prometheus-operated-tls" not found Oct 31 00:35:54.180720 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:54.180693 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a2746a63-fc36-4b4f-a17a-a31e50053bfd-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a2746a63-fc36-4b4f-a17a-a31e50053bfd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:54.180784 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:54.180745 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a2746a63-fc36-4b4f-a17a-a31e50053bfd-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"a2746a63-fc36-4b4f-a17a-a31e50053bfd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:54.180784 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:54.180772 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a2746a63-fc36-4b4f-a17a-a31e50053bfd-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"a2746a63-fc36-4b4f-a17a-a31e50053bfd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:54.181556 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:54.181531 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a2746a63-fc36-4b4f-a17a-a31e50053bfd-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"a2746a63-fc36-4b4f-a17a-a31e50053bfd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:54.183057 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:54.183036 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2746a63-fc36-4b4f-a17a-a31e50053bfd-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a2746a63-fc36-4b4f-a17a-a31e50053bfd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:54.183200 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:54.183179 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a2746a63-fc36-4b4f-a17a-a31e50053bfd-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a2746a63-fc36-4b4f-a17a-a31e50053bfd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:54.183200 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:54.183192 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a2746a63-fc36-4b4f-a17a-a31e50053bfd-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"a2746a63-fc36-4b4f-a17a-a31e50053bfd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:54.192813 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:54.192771 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77c7t\" (UniqueName: \"kubernetes.io/projected/a2746a63-fc36-4b4f-a17a-a31e50053bfd-kube-api-access-77c7t\") pod \"data-science-collector-collector-0\" (UID: \"a2746a63-fc36-4b4f-a17a-a31e50053bfd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:54.591850 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:54.591778 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wdnls" event={"ID":"72eedfa0-5a3d-4047-98ea-c5e9f5582552","Type":"ContainerStarted","Data":"e599d01e41275c0d3668936435933279f22cd23e028ba14707e3e1f590a01ac3"} Oct 31 00:35:54.591850 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:54.591831 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wdnls" event={"ID":"72eedfa0-5a3d-4047-98ea-c5e9f5582552","Type":"ContainerStarted","Data":"5bac39bff688ae2382dcd7423982cd1ffa527758125be94cc4ff7e2250a05855"} Oct 31 00:35:54.592244 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:54.591894 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wdnls" Oct 31 00:35:54.611838 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:54.611793 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wdnls" podStartSLOduration=1.611779794 podStartE2EDuration="1.611779794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:54.61121368 +0000 UTC m=+2191.248032396" watchObservedRunningTime="2026-10-31 00:35:54.611779794 +0000 UTC m=+2191.248598509" Oct 31 00:35:54.686625 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:54.686594 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a2746a63-fc36-4b4f-a17a-a31e50053bfd-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a2746a63-fc36-4b4f-a17a-a31e50053bfd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:54.689153 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:54.689132 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a2746a63-fc36-4b4f-a17a-a31e50053bfd-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a2746a63-fc36-4b4f-a17a-a31e50053bfd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:54.858169 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:54.858103 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:55.132684 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:55.132657 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:35:55.135841 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:35:55.135800 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2746a63_fc36_4b4f_a17a_a31e50053bfd.slice/crio-7e5a80887ab49f3ab25d7db3861390394602b41397bb133a81f6831a4ed4f831 WatchSource:0}: Error finding container 7e5a80887ab49f3ab25d7db3861390394602b41397bb133a81f6831a4ed4f831: Status 404 returned error can't find the container with id 7e5a80887ab49f3ab25d7db3861390394602b41397bb133a81f6831a4ed4f831 Oct 31 00:35:55.138127 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:55.138090 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:35:55.138218 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:55.138148 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:35:55.189639 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:55.189619 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-k28gg\" (UID: \"3dc6465e-bbd3-4e25-b4c6-d28231ef86d6\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k28gg" Oct 31 00:35:55.191649 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:55.191625 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-k28gg\" (UID: \"3dc6465e-bbd3-4e25-b4c6-d28231ef86d6\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k28gg" Oct 31 00:35:55.225783 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:55.225563 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k28gg" Oct 31 00:35:55.501210 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:55.501184 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k28gg"] Oct 31 00:35:55.503739 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:35:55.503712 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dc6465e_bbd3_4e25_b4c6_d28231ef86d6.slice/crio-3097a10d75a789ea4ee45c47b2850c6cc4bc0f7aa34f225ee973efcc27c2c1cc WatchSource:0}: Error finding container 3097a10d75a789ea4ee45c47b2850c6cc4bc0f7aa34f225ee973efcc27c2c1cc: Status 404 returned error can't find the container with id 3097a10d75a789ea4ee45c47b2850c6cc4bc0f7aa34f225ee973efcc27c2c1cc Oct 31 00:35:55.599069 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:55.599040 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"a2746a63-fc36-4b4f-a17a-a31e50053bfd","Type":"ContainerStarted","Data":"a776531851c5100d8b2736e03efb0ede8271cd1bd2adb45c9ea80cb552fcab1b"} Oct 31 00:35:55.599465 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:55.599093 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"a2746a63-fc36-4b4f-a17a-a31e50053bfd","Type":"ContainerStarted","Data":"7e5a80887ab49f3ab25d7db3861390394602b41397bb133a81f6831a4ed4f831"} Oct 31 00:35:55.600369 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:55.600341 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k28gg" event={"ID":"3dc6465e-bbd3-4e25-b4c6-d28231ef86d6","Type":"ContainerStarted","Data":"3097a10d75a789ea4ee45c47b2850c6cc4bc0f7aa34f225ee973efcc27c2c1cc"} Oct 31 00:35:55.622207 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:55.622160 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.6221475229999998 podStartE2EDuration="2.622147523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:55.618671136 +0000 UTC m=+2192.255489853" watchObservedRunningTime="2026-10-31 00:35:55.622147523 +0000 UTC m=+2192.258966236" Oct 31 00:35:56.606548 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:56.606514 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k28gg" event={"ID":"3dc6465e-bbd3-4e25-b4c6-d28231ef86d6","Type":"ContainerStarted","Data":"b06fd2a6db91e81b02491057efef9f2ef2c1e2259d982ae58f91708b94c12f57"} Oct 31 00:35:56.606929 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:56.606583 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k28gg" Oct 31 00:35:56.627486 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:35:56.627441 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k28gg" podStartSLOduration=3.627423674 podStartE2EDuration="3.627423674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:56.626065402 +0000 UTC m=+2193.262884117" watchObservedRunningTime="2026-10-31 00:35:56.627423674 +0000 UTC m=+2193.264242393" Oct 31 00:36:00.398783 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.398745 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:36:00.403383 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.403361 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:36:00.403508 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.403485 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:36:00.407265 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.407249 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:00.410310 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.410290 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 31 00:36:00.413566 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.413527 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 31 00:36:00.413893 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.413873 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 31 00:36:00.414414 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.413998 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 31 00:36:00.415025 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.415004 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 31 00:36:00.416662 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.416632 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:36:00.426593 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.426568 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:36:00.429086 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.429063 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7jqx\" (UniqueName: \"kubernetes.io/projected/1294ed53-4e5a-4105-8acd-b2a04cce8df9-kube-api-access-b7jqx\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"1294ed53-4e5a-4105-8acd-b2a04cce8df9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:36:00.429182 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.429124 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1294ed53-4e5a-4105-8acd-b2a04cce8df9-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"1294ed53-4e5a-4105-8acd-b2a04cce8df9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:36:00.429235 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.429209 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1294ed53-4e5a-4105-8acd-b2a04cce8df9-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"1294ed53-4e5a-4105-8acd-b2a04cce8df9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:36:00.429287 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.429259 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1294ed53-4e5a-4105-8acd-b2a04cce8df9-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"1294ed53-4e5a-4105-8acd-b2a04cce8df9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:36:00.429346 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.429315 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1294ed53-4e5a-4105-8acd-b2a04cce8df9-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"1294ed53-4e5a-4105-8acd-b2a04cce8df9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:36:00.429381 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.429345 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1294ed53-4e5a-4105-8acd-b2a04cce8df9-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"1294ed53-4e5a-4105-8acd-b2a04cce8df9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:36:00.429381 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.429374 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/1294ed53-4e5a-4105-8acd-b2a04cce8df9-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"1294ed53-4e5a-4105-8acd-b2a04cce8df9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:36:00.510897 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.510860 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 31 00:36:00.511096 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.511003 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:36:00.511096 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.511014 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 31 00:36:00.511096 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.511036 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 31 00:36:00.511096 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.511050 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:36:00.511096 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.511090 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:36:00.511342 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.511107 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 31 00:36:00.511342 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.511187 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-dlc5w\"" Oct 31 00:36:00.511342 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.511268 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-pn2f5\"" Oct 31 00:36:00.530578 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.530554 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:00.530663 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.530583 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1294ed53-4e5a-4105-8acd-b2a04cce8df9-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"1294ed53-4e5a-4105-8acd-b2a04cce8df9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:36:00.530663 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.530613 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1294ed53-4e5a-4105-8acd-b2a04cce8df9-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"1294ed53-4e5a-4105-8acd-b2a04cce8df9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:36:00.530736 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.530673 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwnd9\" (UniqueName: \"kubernetes.io/projected/638c279f-efbc-45ec-ac21-5d47e6def796-kube-api-access-gwnd9\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:00.530858 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.530842 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/638c279f-efbc-45ec-ac21-5d47e6def796-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:00.530893 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.530877 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1294ed53-4e5a-4105-8acd-b2a04cce8df9-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"1294ed53-4e5a-4105-8acd-b2a04cce8df9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:36:00.530929 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.530900 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:00.531006 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.530932 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:00.531087 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.531067 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1294ed53-4e5a-4105-8acd-b2a04cce8df9-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"1294ed53-4e5a-4105-8acd-b2a04cce8df9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:36:00.531149 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.531107 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1294ed53-4e5a-4105-8acd-b2a04cce8df9-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"1294ed53-4e5a-4105-8acd-b2a04cce8df9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:36:00.531149 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.531138 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/1294ed53-4e5a-4105-8acd-b2a04cce8df9-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"1294ed53-4e5a-4105-8acd-b2a04cce8df9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:36:00.531257 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.531183 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/638c279f-efbc-45ec-ac21-5d47e6def796-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:00.531317 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.531280 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:00.531412 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.531342 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/638c279f-efbc-45ec-ac21-5d47e6def796-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:00.531412 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.531404 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7jqx\" (UniqueName: \"kubernetes.io/projected/1294ed53-4e5a-4105-8acd-b2a04cce8df9-kube-api-access-b7jqx\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"1294ed53-4e5a-4105-8acd-b2a04cce8df9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:36:00.531572 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.531447 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:00.531572 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.531490 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:00.531572 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.531538 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/638c279f-efbc-45ec-ac21-5d47e6def796-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:00.531748 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.531633 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/638c279f-efbc-45ec-ac21-5d47e6def796-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:00.531911 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.531865 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/1294ed53-4e5a-4105-8acd-b2a04cce8df9-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"1294ed53-4e5a-4105-8acd-b2a04cce8df9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:36:00.533599 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.533580 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1294ed53-4e5a-4105-8acd-b2a04cce8df9-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"1294ed53-4e5a-4105-8acd-b2a04cce8df9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:36:00.533741 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.533724 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1294ed53-4e5a-4105-8acd-b2a04cce8df9-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"1294ed53-4e5a-4105-8acd-b2a04cce8df9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:36:00.533819 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.533801 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1294ed53-4e5a-4105-8acd-b2a04cce8df9-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"1294ed53-4e5a-4105-8acd-b2a04cce8df9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:36:00.533864 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.533810 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1294ed53-4e5a-4105-8acd-b2a04cce8df9-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"1294ed53-4e5a-4105-8acd-b2a04cce8df9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:36:00.534117 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.534097 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1294ed53-4e5a-4105-8acd-b2a04cce8df9-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"1294ed53-4e5a-4105-8acd-b2a04cce8df9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:36:00.544183 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.544166 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7jqx\" (UniqueName: \"kubernetes.io/projected/1294ed53-4e5a-4105-8acd-b2a04cce8df9-kube-api-access-b7jqx\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"1294ed53-4e5a-4105-8acd-b2a04cce8df9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:36:00.632628 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.632605 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/638c279f-efbc-45ec-ac21-5d47e6def796-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:00.632747 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.632642 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:00.632747 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.632669 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwnd9\" (UniqueName: \"kubernetes.io/projected/638c279f-efbc-45ec-ac21-5d47e6def796-kube-api-access-gwnd9\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:00.632747 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.632686 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/638c279f-efbc-45ec-ac21-5d47e6def796-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:00.632747 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.632716 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:00.632747 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.632735 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:00.633051 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.632774 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/638c279f-efbc-45ec-ac21-5d47e6def796-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:00.633051 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.632801 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:00.633051 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.632999 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/638c279f-efbc-45ec-ac21-5d47e6def796-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:00.633204 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.633055 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:00.633204 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.633082 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:00.633204 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.633116 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/638c279f-efbc-45ec-ac21-5d47e6def796-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:00.633391 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.633362 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/638c279f-efbc-45ec-ac21-5d47e6def796-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:00.633941 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.633909 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/638c279f-efbc-45ec-ac21-5d47e6def796-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:00.634103 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.633986 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/638c279f-efbc-45ec-ac21-5d47e6def796-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:00.635727 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.635661 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:00.635727 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.635671 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/638c279f-efbc-45ec-ac21-5d47e6def796-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:00.635902 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.635873 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:00.636014 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.635988 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:00.636075 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.636043 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:00.636137 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.636104 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/638c279f-efbc-45ec-ac21-5d47e6def796-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:00.637332 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.637316 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:00.637619 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.637605 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:00.645163 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.645143 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwnd9\" (UniqueName: \"kubernetes.io/projected/638c279f-efbc-45ec-ac21-5d47e6def796-kube-api-access-gwnd9\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:00.725910 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:00.725858 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:36:01.011689 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:01.007549 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:36:01.633293 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:01.633256 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"1294ed53-4e5a-4105-8acd-b2a04cce8df9","Type":"ContainerStarted","Data":"d9a54253cea9c7ef6818bc231a5416a8c09dbcd5294c96a097b3f3f3dbda105c"} Oct 31 00:36:01.633679 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:01.633328 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"1294ed53-4e5a-4105-8acd-b2a04cce8df9","Type":"ContainerStarted","Data":"132d06a037a86cdaade8a9f9e97cdd4b17d1dfaf9cf697935409ece0e5dcfe01"} Oct 31 00:36:02.614180 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:02.614149 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k28gg" Oct 31 00:36:05.602613 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:05.602584 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wdnls" Oct 31 00:36:05.654323 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:05.654291 5274 generic.go:356] "Generic (PLEG): container finished" podID="1294ed53-4e5a-4105-8acd-b2a04cce8df9" containerID="d9a54253cea9c7ef6818bc231a5416a8c09dbcd5294c96a097b3f3f3dbda105c" exitCode=0 Oct 31 00:36:05.654484 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:05.654363 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"1294ed53-4e5a-4105-8acd-b2a04cce8df9","Type":"ContainerDied","Data":"d9a54253cea9c7ef6818bc231a5416a8c09dbcd5294c96a097b3f3f3dbda105c"} Oct 31 00:36:06.661736 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:06.661695 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"1294ed53-4e5a-4105-8acd-b2a04cce8df9","Type":"ContainerStarted","Data":"74f50f1c5d3b2e1f48497e1aee716fbc62b70202794e48526ecf56578c8a6837"} Oct 31 00:36:06.662229 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:06.661784 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"1294ed53-4e5a-4105-8acd-b2a04cce8df9","Type":"ContainerStarted","Data":"8336c41d20990947f1b50c861028d66de9cc789fdb475a52a5d50d2e6033a51b"} Oct 31 00:36:06.662229 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:06.661995 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:36:06.689519 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:06.689475 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=3.304705974 podStartE2EDuration="6.689464026s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.384758052s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:06.688375336 +0000 UTC m=+2203.325194053" watchObservedRunningTime="2026-10-31 00:36:06.689464026 +0000 UTC m=+2203.326282742" Oct 31 00:36:07.692575 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:07.692539 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:07.699859 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:07.699839 5274 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:07.793527 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:07.793500 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:07.793641 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:07.793592 5274 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:07.797142 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:07.797125 5274 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") DevicePath \"csi-51e8027f163b5f98040faf3f8c30848dd44a7a90ca2bd0af2446ea6398ed018d\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:07.886897 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:07.886865 5274 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/53bcf0e8434a816dcc013cea20657add42b4da36ecdc0fff46af7413bc0037a5/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:07.901602 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:07.901581 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:07.935592 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:07.935570 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:08.245861 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:08.245832 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:36:08.247377 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:36:08.247339 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod638c279f_efbc_45ec_ac21_5d47e6def796.slice/crio-6a39b169d778ef0e23ea7c0b5afa1f3b24e133c4714cf7432018ce5f90f84d6c WatchSource:0}: Error finding container 6a39b169d778ef0e23ea7c0b5afa1f3b24e133c4714cf7432018ce5f90f84d6c: Status 404 returned error can't find the container with id 6a39b169d778ef0e23ea7c0b5afa1f3b24e133c4714cf7432018ce5f90f84d6c Oct 31 00:36:08.673772 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:08.673745 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"638c279f-efbc-45ec-ac21-5d47e6def796","Type":"ContainerStarted","Data":"47de2c4275cc951b65c9360d45be4b57ab1bf2ce5a8c29d55731c98b95b5c5e7"} Oct 31 00:36:08.673887 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:08.673812 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"638c279f-efbc-45ec-ac21-5d47e6def796","Type":"ContainerStarted","Data":"6a39b169d778ef0e23ea7c0b5afa1f3b24e133c4714cf7432018ce5f90f84d6c"} Oct 31 00:36:12.667974 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:12.667927 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:36:12.695040 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:12.695004 5274 generic.go:356] "Generic (PLEG): container finished" podID="638c279f-efbc-45ec-ac21-5d47e6def796" containerID="47de2c4275cc951b65c9360d45be4b57ab1bf2ce5a8c29d55731c98b95b5c5e7" exitCode=0 Oct 31 00:36:12.695298 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:12.695039 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"638c279f-efbc-45ec-ac21-5d47e6def796","Type":"ContainerDied","Data":"47de2c4275cc951b65c9360d45be4b57ab1bf2ce5a8c29d55731c98b95b5c5e7"} Oct 31 00:36:13.703128 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:13.703091 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"638c279f-efbc-45ec-ac21-5d47e6def796","Type":"ContainerStarted","Data":"2b24cbad42e3d9e26b15f0c49f8bbd9c7f6be7c5ff9b4770d486552f0e95db4a"} Oct 31 00:36:13.703625 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:13.703172 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"638c279f-efbc-45ec-ac21-5d47e6def796","Type":"ContainerStarted","Data":"e8e42586c9f22feb9b15b05543e5538500b873b0c2c889add6cec90c662cd0da"} Oct 31 00:36:13.703625 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:13.703210 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"638c279f-efbc-45ec-ac21-5d47e6def796","Type":"ContainerStarted","Data":"71fa5393b769366c416449f82bd3143db69a43e18de39b3d7f9c9d831540a48a"} Oct 31 00:36:13.736678 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:13.736632 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=10.251727202 podStartE2EDuration="13.736620824s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.484893622s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:13.734223643 +0000 UTC m=+2210.371042360" watchObservedRunningTime="2026-10-31 00:36:13.736620824 +0000 UTC m=+2210.373439534" Oct 31 00:36:17.935837 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:17.935806 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:20.898725 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:36:20.898688 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:36:22.936589 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:22.936555 5274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:22.941793 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:22.941772 5274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:23.763913 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:23.763886 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:36:35.937509 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:35.937476 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:36:35.938109 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:35.937788 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="a2746a63-fc36-4b4f-a17a-a31e50053bfd" containerName="otc-container" containerID="cri-o://a776531851c5100d8b2736e03efb0ede8271cd1bd2adb45c9ea80cb552fcab1b" gracePeriod=30 Oct 31 00:36:38.281045 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.281019 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:36:38.347830 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.347747 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2746a63-fc36-4b4f-a17a-a31e50053bfd-otc-internal\" (UniqueName: \"kubernetes.io/configmap/a2746a63-fc36-4b4f-a17a-a31e50053bfd-otc-internal\") pod \"a2746a63-fc36-4b4f-a17a-a31e50053bfd\" (UID: \"a2746a63-fc36-4b4f-a17a-a31e50053bfd\") " Oct 31 00:36:38.347830 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.347797 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2746a63-fc36-4b4f-a17a-a31e50053bfd-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a2746a63-fc36-4b4f-a17a-a31e50053bfd-otlp-tls-certs\") pod \"a2746a63-fc36-4b4f-a17a-a31e50053bfd\" (UID: \"a2746a63-fc36-4b4f-a17a-a31e50053bfd\") " Oct 31 00:36:38.348090 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.347887 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2746a63-fc36-4b4f-a17a-a31e50053bfd-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a2746a63-fc36-4b4f-a17a-a31e50053bfd-data-science-collector-ta-client-cert\") pod \"a2746a63-fc36-4b4f-a17a-a31e50053bfd\" (UID: \"a2746a63-fc36-4b4f-a17a-a31e50053bfd\") " Oct 31 00:36:38.348090 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.348002 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2746a63-fc36-4b4f-a17a-a31e50053bfd-kube-api-access-77c7t\" (UniqueName: \"kubernetes.io/projected/a2746a63-fc36-4b4f-a17a-a31e50053bfd-kube-api-access-77c7t\") pod \"a2746a63-fc36-4b4f-a17a-a31e50053bfd\" (UID: \"a2746a63-fc36-4b4f-a17a-a31e50053bfd\") " Oct 31 00:36:38.348213 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.348171 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2746a63-fc36-4b4f-a17a-a31e50053bfd-otc-internal" pod "a2746a63-fc36-4b4f-a17a-a31e50053bfd" (UID: "a2746a63-fc36-4b4f-a17a-a31e50053bfd"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:36:38.348268 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.348037 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2746a63-fc36-4b4f-a17a-a31e50053bfd-tls-certs\" (UniqueName: \"kubernetes.io/secret/a2746a63-fc36-4b4f-a17a-a31e50053bfd-tls-certs\") pod \"a2746a63-fc36-4b4f-a17a-a31e50053bfd\" (UID: \"a2746a63-fc36-4b4f-a17a-a31e50053bfd\") " Oct 31 00:36:38.348268 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.348260 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2746a63-fc36-4b4f-a17a-a31e50053bfd-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a2746a63-fc36-4b4f-a17a-a31e50053bfd-monitoring-tls-certs\") pod \"a2746a63-fc36-4b4f-a17a-a31e50053bfd\" (UID: \"a2746a63-fc36-4b4f-a17a-a31e50053bfd\") " Oct 31 00:36:38.348500 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.348480 5274 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a2746a63-fc36-4b4f-a17a-a31e50053bfd-otc-internal\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:38.350219 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.350196 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2746a63-fc36-4b4f-a17a-a31e50053bfd-otlp-tls-certs" pod "a2746a63-fc36-4b4f-a17a-a31e50053bfd" (UID: "a2746a63-fc36-4b4f-a17a-a31e50053bfd"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:36:38.350351 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.350331 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2746a63-fc36-4b4f-a17a-a31e50053bfd-data-science-collector-ta-client-cert" pod "a2746a63-fc36-4b4f-a17a-a31e50053bfd" (UID: "a2746a63-fc36-4b4f-a17a-a31e50053bfd"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:36:38.350427 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.350363 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2746a63-fc36-4b4f-a17a-a31e50053bfd-kube-api-access-77c7t" pod "a2746a63-fc36-4b4f-a17a-a31e50053bfd" (UID: "a2746a63-fc36-4b4f-a17a-a31e50053bfd"). InnerVolumeSpecName "kube-api-access-77c7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:38.350625 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.350598 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2746a63-fc36-4b4f-a17a-a31e50053bfd-tls-certs" pod "a2746a63-fc36-4b4f-a17a-a31e50053bfd" (UID: "a2746a63-fc36-4b4f-a17a-a31e50053bfd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:36:38.350714 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.350616 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2746a63-fc36-4b4f-a17a-a31e50053bfd-monitoring-tls-certs" pod "a2746a63-fc36-4b4f-a17a-a31e50053bfd" (UID: "a2746a63-fc36-4b4f-a17a-a31e50053bfd"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:36:38.449121 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.449094 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77c7t\" (UniqueName: \"kubernetes.io/projected/a2746a63-fc36-4b4f-a17a-a31e50053bfd-kube-api-access-77c7t\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:38.449121 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.449120 5274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2746a63-fc36-4b4f-a17a-a31e50053bfd-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:38.449276 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.449130 5274 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a2746a63-fc36-4b4f-a17a-a31e50053bfd-monitoring-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:38.449276 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.449139 5274 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a2746a63-fc36-4b4f-a17a-a31e50053bfd-otlp-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:38.449276 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.449149 5274 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a2746a63-fc36-4b4f-a17a-a31e50053bfd-data-science-collector-ta-client-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:38.835916 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.835885 5274 generic.go:356] "Generic (PLEG): container finished" podID="a2746a63-fc36-4b4f-a17a-a31e50053bfd" containerID="a776531851c5100d8b2736e03efb0ede8271cd1bd2adb45c9ea80cb552fcab1b" exitCode=0 Oct 31 00:36:38.836110 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.835933 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:36:38.836110 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.835986 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"a2746a63-fc36-4b4f-a17a-a31e50053bfd","Type":"ContainerDied","Data":"a776531851c5100d8b2736e03efb0ede8271cd1bd2adb45c9ea80cb552fcab1b"} Oct 31 00:36:38.836110 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.836040 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"a2746a63-fc36-4b4f-a17a-a31e50053bfd","Type":"ContainerDied","Data":"7e5a80887ab49f3ab25d7db3861390394602b41397bb133a81f6831a4ed4f831"} Oct 31 00:36:38.836110 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.836062 5274 scope.go:122] "RemoveContainer" containerID="a776531851c5100d8b2736e03efb0ede8271cd1bd2adb45c9ea80cb552fcab1b" Oct 31 00:36:38.845635 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.845621 5274 scope.go:122] "RemoveContainer" containerID="a776531851c5100d8b2736e03efb0ede8271cd1bd2adb45c9ea80cb552fcab1b" Oct 31 00:36:38.845915 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:36:38.845896 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a776531851c5100d8b2736e03efb0ede8271cd1bd2adb45c9ea80cb552fcab1b\": container with ID starting with a776531851c5100d8b2736e03efb0ede8271cd1bd2adb45c9ea80cb552fcab1b not found: ID does not exist" containerID="a776531851c5100d8b2736e03efb0ede8271cd1bd2adb45c9ea80cb552fcab1b" Oct 31 00:36:38.845997 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.845922 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a776531851c5100d8b2736e03efb0ede8271cd1bd2adb45c9ea80cb552fcab1b"} err="failed to get container status \"a776531851c5100d8b2736e03efb0ede8271cd1bd2adb45c9ea80cb552fcab1b\": rpc error: code = NotFound desc = could not find container \"a776531851c5100d8b2736e03efb0ede8271cd1bd2adb45c9ea80cb552fcab1b\": container with ID starting with a776531851c5100d8b2736e03efb0ede8271cd1bd2adb45c9ea80cb552fcab1b not found: ID does not exist" Oct 31 00:36:38.857999 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.857978 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:36:38.862875 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.862855 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:36:38.887699 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.887678 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:36:38.888088 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.888075 5274 cpu_manager.go:414] "containerMap: removing container" podUID="a2746a63-fc36-4b4f-a17a-a31e50053bfd" containerName="otc-container" Oct 31 00:36:38.888130 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.888090 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2746a63-fc36-4b4f-a17a-a31e50053bfd" containerName="otc-container" Oct 31 00:36:38.888179 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.888170 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2746a63-fc36-4b4f-a17a-a31e50053bfd" containerName="otc-container" Oct 31 00:36:38.892829 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.892812 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:36:38.902088 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.902062 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:36:38.902218 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.902204 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:36:38.902294 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.902277 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:36:38.902414 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.902342 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-cfb2n\"" Oct 31 00:36:38.902483 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.902430 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:36:38.902544 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.902520 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:36:38.904093 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.904071 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:36:38.953785 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.953757 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7216f712-5ed6-456d-bf6b-40c326ce28aa-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7216f712-5ed6-456d-bf6b-40c326ce28aa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:36:38.953936 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.953801 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7216f712-5ed6-456d-bf6b-40c326ce28aa-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"7216f712-5ed6-456d-bf6b-40c326ce28aa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:36:38.953936 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.953827 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xngf\" (UniqueName: \"kubernetes.io/projected/7216f712-5ed6-456d-bf6b-40c326ce28aa-kube-api-access-6xngf\") pod \"data-science-collector-collector-0\" (UID: \"7216f712-5ed6-456d-bf6b-40c326ce28aa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:36:38.954126 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.954015 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7216f712-5ed6-456d-bf6b-40c326ce28aa-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"7216f712-5ed6-456d-bf6b-40c326ce28aa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:36:38.954126 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.954050 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7216f712-5ed6-456d-bf6b-40c326ce28aa-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7216f712-5ed6-456d-bf6b-40c326ce28aa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:36:38.954126 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:38.954119 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7216f712-5ed6-456d-bf6b-40c326ce28aa-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7216f712-5ed6-456d-bf6b-40c326ce28aa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:36:39.054752 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:39.054718 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7216f712-5ed6-456d-bf6b-40c326ce28aa-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"7216f712-5ed6-456d-bf6b-40c326ce28aa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:36:39.054752 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:39.054752 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7216f712-5ed6-456d-bf6b-40c326ce28aa-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7216f712-5ed6-456d-bf6b-40c326ce28aa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:36:39.055012 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:39.054793 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7216f712-5ed6-456d-bf6b-40c326ce28aa-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7216f712-5ed6-456d-bf6b-40c326ce28aa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:36:39.055012 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:39.054828 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7216f712-5ed6-456d-bf6b-40c326ce28aa-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7216f712-5ed6-456d-bf6b-40c326ce28aa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:36:39.055012 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:39.054869 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7216f712-5ed6-456d-bf6b-40c326ce28aa-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"7216f712-5ed6-456d-bf6b-40c326ce28aa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:36:39.055012 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:39.054892 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xngf\" (UniqueName: \"kubernetes.io/projected/7216f712-5ed6-456d-bf6b-40c326ce28aa-kube-api-access-6xngf\") pod \"data-science-collector-collector-0\" (UID: \"7216f712-5ed6-456d-bf6b-40c326ce28aa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:36:39.055636 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:39.055605 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7216f712-5ed6-456d-bf6b-40c326ce28aa-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"7216f712-5ed6-456d-bf6b-40c326ce28aa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:36:39.057387 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:39.057361 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7216f712-5ed6-456d-bf6b-40c326ce28aa-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7216f712-5ed6-456d-bf6b-40c326ce28aa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:36:39.057471 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:39.057418 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7216f712-5ed6-456d-bf6b-40c326ce28aa-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"7216f712-5ed6-456d-bf6b-40c326ce28aa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:36:39.057745 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:39.057724 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7216f712-5ed6-456d-bf6b-40c326ce28aa-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7216f712-5ed6-456d-bf6b-40c326ce28aa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:36:39.057785 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:39.057733 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7216f712-5ed6-456d-bf6b-40c326ce28aa-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7216f712-5ed6-456d-bf6b-40c326ce28aa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:36:39.066813 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:39.066792 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xngf\" (UniqueName: \"kubernetes.io/projected/7216f712-5ed6-456d-bf6b-40c326ce28aa-kube-api-access-6xngf\") pod \"data-science-collector-collector-0\" (UID: \"7216f712-5ed6-456d-bf6b-40c326ce28aa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:36:39.205625 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:39.205603 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:36:39.483090 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:36:39.483064 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7216f712_5ed6_456d_bf6b_40c326ce28aa.slice/crio-04358327ae00bbabeaf8fb22f4e2185b8b96120a5357a6e605866f184b81ee81 WatchSource:0}: Error finding container 04358327ae00bbabeaf8fb22f4e2185b8b96120a5357a6e605866f184b81ee81: Status 404 returned error can't find the container with id 04358327ae00bbabeaf8fb22f4e2185b8b96120a5357a6e605866f184b81ee81 Oct 31 00:36:39.483394 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:39.483369 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2746a63-fc36-4b4f-a17a-a31e50053bfd" path="/var/lib/kubelet/pods/a2746a63-fc36-4b4f-a17a-a31e50053bfd/volumes" Oct 31 00:36:39.483789 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:39.483765 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:36:39.485096 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:39.485068 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:36:39.485175 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:39.485123 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:36:39.842492 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:39.842407 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"7216f712-5ed6-456d-bf6b-40c326ce28aa","Type":"ContainerStarted","Data":"9ebb1b753858dc2984bde65a2175009a6900a03ef57f1c3c57222f430a48cf15"} Oct 31 00:36:39.842492 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:39.842462 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"7216f712-5ed6-456d-bf6b-40c326ce28aa","Type":"ContainerStarted","Data":"04358327ae00bbabeaf8fb22f4e2185b8b96120a5357a6e605866f184b81ee81"} Oct 31 00:36:39.864502 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:36:39.864456 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.86444358 podStartE2EDuration="1.86444358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:39.862897023 +0000 UTC m=+2236.499715740" watchObservedRunningTime="2026-10-31 00:36:39.86444358 +0000 UTC m=+2236.501262295" Oct 31 00:36:50.900043 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:36:50.900002 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:37:09.947011 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:09.946873 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-r2sfh"] Oct 31 00:37:09.950807 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:09.950785 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-r2sfh" Oct 31 00:37:09.960785 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:09.960755 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-r2sfh"] Oct 31 00:37:10.111785 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:10.111755 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n6xr\" (UniqueName: \"kubernetes.io/projected/9f61d0fb-44a1-4c82-a9ca-ef596732f2ba-kube-api-access-2n6xr\") pod \"korrel8r-66d6f7f696-r2sfh\" (UID: \"9f61d0fb-44a1-4c82-a9ca-ef596732f2ba\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-r2sfh" Oct 31 00:37:10.111976 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:10.111808 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9f61d0fb-44a1-4c82-a9ca-ef596732f2ba-config\") pod \"korrel8r-66d6f7f696-r2sfh\" (UID: \"9f61d0fb-44a1-4c82-a9ca-ef596732f2ba\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-r2sfh" Oct 31 00:37:10.111976 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:10.111896 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/9f61d0fb-44a1-4c82-a9ca-ef596732f2ba-tls\") pod \"korrel8r-66d6f7f696-r2sfh\" (UID: \"9f61d0fb-44a1-4c82-a9ca-ef596732f2ba\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-r2sfh" Oct 31 00:37:10.212644 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:10.212577 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n6xr\" (UniqueName: \"kubernetes.io/projected/9f61d0fb-44a1-4c82-a9ca-ef596732f2ba-kube-api-access-2n6xr\") pod \"korrel8r-66d6f7f696-r2sfh\" (UID: \"9f61d0fb-44a1-4c82-a9ca-ef596732f2ba\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-r2sfh" Oct 31 00:37:10.212644 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:10.212624 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9f61d0fb-44a1-4c82-a9ca-ef596732f2ba-config\") pod \"korrel8r-66d6f7f696-r2sfh\" (UID: \"9f61d0fb-44a1-4c82-a9ca-ef596732f2ba\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-r2sfh" Oct 31 00:37:10.212864 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:10.212651 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/9f61d0fb-44a1-4c82-a9ca-ef596732f2ba-tls\") pod \"korrel8r-66d6f7f696-r2sfh\" (UID: \"9f61d0fb-44a1-4c82-a9ca-ef596732f2ba\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-r2sfh" Oct 31 00:37:10.213420 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:10.213395 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9f61d0fb-44a1-4c82-a9ca-ef596732f2ba-config\") pod \"korrel8r-66d6f7f696-r2sfh\" (UID: \"9f61d0fb-44a1-4c82-a9ca-ef596732f2ba\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-r2sfh" Oct 31 00:37:10.215132 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:10.215109 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/9f61d0fb-44a1-4c82-a9ca-ef596732f2ba-tls\") pod \"korrel8r-66d6f7f696-r2sfh\" (UID: \"9f61d0fb-44a1-4c82-a9ca-ef596732f2ba\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-r2sfh" Oct 31 00:37:10.225967 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:10.225931 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n6xr\" (UniqueName: \"kubernetes.io/projected/9f61d0fb-44a1-4c82-a9ca-ef596732f2ba-kube-api-access-2n6xr\") pod \"korrel8r-66d6f7f696-r2sfh\" (UID: \"9f61d0fb-44a1-4c82-a9ca-ef596732f2ba\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-r2sfh" Oct 31 00:37:10.264909 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:10.264886 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-r2sfh" Oct 31 00:37:10.546281 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:10.546256 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-r2sfh"] Oct 31 00:37:10.548312 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:37:10.548276 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f61d0fb_44a1_4c82_a9ca_ef596732f2ba.slice/crio-f697734b18e89afe74b4709263efdcdc0eda7858e79da48dfefe43ea99cf8600 WatchSource:0}: Error finding container f697734b18e89afe74b4709263efdcdc0eda7858e79da48dfefe43ea99cf8600: Status 404 returned error can't find the container with id f697734b18e89afe74b4709263efdcdc0eda7858e79da48dfefe43ea99cf8600 Oct 31 00:37:10.999653 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:10.999612 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-r2sfh" event={"ID":"9f61d0fb-44a1-4c82-a9ca-ef596732f2ba","Type":"ContainerStarted","Data":"25a1e71fea134e8d15519976ba3a3460f472625be38b7637316a542d666753dd"} Oct 31 00:37:11.000064 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:10.999675 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-r2sfh" event={"ID":"9f61d0fb-44a1-4c82-a9ca-ef596732f2ba","Type":"ContainerStarted","Data":"f697734b18e89afe74b4709263efdcdc0eda7858e79da48dfefe43ea99cf8600"} Oct 31 00:37:11.000064 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:10.999703 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-r2sfh" Oct 31 00:37:11.019048 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:11.018974 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-r2sfh" podStartSLOduration=2.018939434 podStartE2EDuration="2.018939434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:11.018051489 +0000 UTC m=+2267.654870219" watchObservedRunningTime="2026-10-31 00:37:11.018939434 +0000 UTC m=+2267.655758170" Oct 31 00:37:14.647123 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:14.647089 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:37:14.647611 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:14.647281 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="7216f712-5ed6-456d-bf6b-40c326ce28aa" containerName="otc-container" containerID="cri-o://9ebb1b753858dc2984bde65a2175009a6900a03ef57f1c3c57222f430a48cf15" gracePeriod=30 Oct 31 00:37:16.993318 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:16.993295 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:17.030835 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.030803 5274 generic.go:356] "Generic (PLEG): container finished" podID="7216f712-5ed6-456d-bf6b-40c326ce28aa" containerID="9ebb1b753858dc2984bde65a2175009a6900a03ef57f1c3c57222f430a48cf15" exitCode=0 Oct 31 00:37:17.030986 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.030849 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:17.030986 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.030884 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"7216f712-5ed6-456d-bf6b-40c326ce28aa","Type":"ContainerDied","Data":"9ebb1b753858dc2984bde65a2175009a6900a03ef57f1c3c57222f430a48cf15"} Oct 31 00:37:17.030986 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.030934 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"7216f712-5ed6-456d-bf6b-40c326ce28aa","Type":"ContainerDied","Data":"04358327ae00bbabeaf8fb22f4e2185b8b96120a5357a6e605866f184b81ee81"} Oct 31 00:37:17.030986 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.030974 5274 scope.go:122] "RemoveContainer" containerID="9ebb1b753858dc2984bde65a2175009a6900a03ef57f1c3c57222f430a48cf15" Oct 31 00:37:17.039665 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.039644 5274 scope.go:122] "RemoveContainer" containerID="9ebb1b753858dc2984bde65a2175009a6900a03ef57f1c3c57222f430a48cf15" Oct 31 00:37:17.039920 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:37:17.039899 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ebb1b753858dc2984bde65a2175009a6900a03ef57f1c3c57222f430a48cf15\": container with ID starting with 9ebb1b753858dc2984bde65a2175009a6900a03ef57f1c3c57222f430a48cf15 not found: ID does not exist" containerID="9ebb1b753858dc2984bde65a2175009a6900a03ef57f1c3c57222f430a48cf15" Oct 31 00:37:17.040010 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.039933 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ebb1b753858dc2984bde65a2175009a6900a03ef57f1c3c57222f430a48cf15"} err="failed to get container status \"9ebb1b753858dc2984bde65a2175009a6900a03ef57f1c3c57222f430a48cf15\": rpc error: code = NotFound desc = could not find container \"9ebb1b753858dc2984bde65a2175009a6900a03ef57f1c3c57222f430a48cf15\": container with ID starting with 9ebb1b753858dc2984bde65a2175009a6900a03ef57f1c3c57222f430a48cf15 not found: ID does not exist" Oct 31 00:37:17.177573 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.177540 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7216f712-5ed6-456d-bf6b-40c326ce28aa-kube-api-access-6xngf\" (UniqueName: \"kubernetes.io/projected/7216f712-5ed6-456d-bf6b-40c326ce28aa-kube-api-access-6xngf\") pod \"7216f712-5ed6-456d-bf6b-40c326ce28aa\" (UID: \"7216f712-5ed6-456d-bf6b-40c326ce28aa\") " Oct 31 00:37:17.177752 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.177591 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7216f712-5ed6-456d-bf6b-40c326ce28aa-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7216f712-5ed6-456d-bf6b-40c326ce28aa-monitoring-tls-certs\") pod \"7216f712-5ed6-456d-bf6b-40c326ce28aa\" (UID: \"7216f712-5ed6-456d-bf6b-40c326ce28aa\") " Oct 31 00:37:17.177752 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.177647 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7216f712-5ed6-456d-bf6b-40c326ce28aa-otc-internal\" (UniqueName: \"kubernetes.io/configmap/7216f712-5ed6-456d-bf6b-40c326ce28aa-otc-internal\") pod \"7216f712-5ed6-456d-bf6b-40c326ce28aa\" (UID: \"7216f712-5ed6-456d-bf6b-40c326ce28aa\") " Oct 31 00:37:17.177752 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.177692 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7216f712-5ed6-456d-bf6b-40c326ce28aa-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7216f712-5ed6-456d-bf6b-40c326ce28aa-otlp-tls-certs\") pod \"7216f712-5ed6-456d-bf6b-40c326ce28aa\" (UID: \"7216f712-5ed6-456d-bf6b-40c326ce28aa\") " Oct 31 00:37:17.177752 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.177721 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7216f712-5ed6-456d-bf6b-40c326ce28aa-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7216f712-5ed6-456d-bf6b-40c326ce28aa-data-science-collector-ta-client-cert\") pod \"7216f712-5ed6-456d-bf6b-40c326ce28aa\" (UID: \"7216f712-5ed6-456d-bf6b-40c326ce28aa\") " Oct 31 00:37:17.177997 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.177778 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7216f712-5ed6-456d-bf6b-40c326ce28aa-tls-certs\" (UniqueName: \"kubernetes.io/secret/7216f712-5ed6-456d-bf6b-40c326ce28aa-tls-certs\") pod \"7216f712-5ed6-456d-bf6b-40c326ce28aa\" (UID: \"7216f712-5ed6-456d-bf6b-40c326ce28aa\") " Oct 31 00:37:17.178140 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.178109 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7216f712-5ed6-456d-bf6b-40c326ce28aa-otc-internal" pod "7216f712-5ed6-456d-bf6b-40c326ce28aa" (UID: "7216f712-5ed6-456d-bf6b-40c326ce28aa"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:17.179879 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.179846 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7216f712-5ed6-456d-bf6b-40c326ce28aa-data-science-collector-ta-client-cert" pod "7216f712-5ed6-456d-bf6b-40c326ce28aa" (UID: "7216f712-5ed6-456d-bf6b-40c326ce28aa"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:17.180009 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.179856 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7216f712-5ed6-456d-bf6b-40c326ce28aa-otlp-tls-certs" pod "7216f712-5ed6-456d-bf6b-40c326ce28aa" (UID: "7216f712-5ed6-456d-bf6b-40c326ce28aa"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:17.180009 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.179991 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7216f712-5ed6-456d-bf6b-40c326ce28aa-kube-api-access-6xngf" pod "7216f712-5ed6-456d-bf6b-40c326ce28aa" (UID: "7216f712-5ed6-456d-bf6b-40c326ce28aa"). InnerVolumeSpecName "kube-api-access-6xngf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:17.180262 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.180241 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7216f712-5ed6-456d-bf6b-40c326ce28aa-tls-certs" pod "7216f712-5ed6-456d-bf6b-40c326ce28aa" (UID: "7216f712-5ed6-456d-bf6b-40c326ce28aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:17.180325 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.180264 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7216f712-5ed6-456d-bf6b-40c326ce28aa-monitoring-tls-certs" pod "7216f712-5ed6-456d-bf6b-40c326ce28aa" (UID: "7216f712-5ed6-456d-bf6b-40c326ce28aa"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:17.279444 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.279425 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xngf\" (UniqueName: \"kubernetes.io/projected/7216f712-5ed6-456d-bf6b-40c326ce28aa-kube-api-access-6xngf\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:17.279444 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.279446 5274 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7216f712-5ed6-456d-bf6b-40c326ce28aa-monitoring-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:17.279592 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.279456 5274 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7216f712-5ed6-456d-bf6b-40c326ce28aa-otc-internal\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:17.279592 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.279465 5274 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7216f712-5ed6-456d-bf6b-40c326ce28aa-otlp-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:17.279592 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.279474 5274 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7216f712-5ed6-456d-bf6b-40c326ce28aa-data-science-collector-ta-client-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:17.279592 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.279484 5274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7216f712-5ed6-456d-bf6b-40c326ce28aa-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:17.353190 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.353166 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:37:17.357964 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.357922 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:37:17.383756 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.383732 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:37:17.384136 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.384122 5274 cpu_manager.go:414] "containerMap: removing container" podUID="7216f712-5ed6-456d-bf6b-40c326ce28aa" containerName="otc-container" Oct 31 00:37:17.384190 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.384138 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7216f712-5ed6-456d-bf6b-40c326ce28aa" containerName="otc-container" Oct 31 00:37:17.384225 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.384219 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="7216f712-5ed6-456d-bf6b-40c326ce28aa" containerName="otc-container" Oct 31 00:37:17.388717 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.388699 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:17.399693 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.399669 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:37:17.399824 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.399796 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-6d9d4583\"" Oct 31 00:37:17.399896 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.399852 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-cfb2n\"" Oct 31 00:37:17.399896 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.399885 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:37:17.400044 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.399942 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:37:17.400112 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.400053 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:37:17.402553 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.402381 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:37:17.481619 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.481594 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84jvr\" (UniqueName: \"kubernetes.io/projected/f31a1848-2cd1-4c98-913e-f567e6d02f1a-kube-api-access-84jvr\") pod \"data-science-collector-collector-0\" (UID: \"f31a1848-2cd1-4c98-913e-f567e6d02f1a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:17.481747 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.481625 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f31a1848-2cd1-4c98-913e-f567e6d02f1a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"f31a1848-2cd1-4c98-913e-f567e6d02f1a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:17.481747 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.481697 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f31a1848-2cd1-4c98-913e-f567e6d02f1a-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"f31a1848-2cd1-4c98-913e-f567e6d02f1a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:17.481833 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.481748 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f31a1848-2cd1-4c98-913e-f567e6d02f1a-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"f31a1848-2cd1-4c98-913e-f567e6d02f1a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:17.481833 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.481790 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f31a1848-2cd1-4c98-913e-f567e6d02f1a-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"f31a1848-2cd1-4c98-913e-f567e6d02f1a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:17.481904 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.481876 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f31a1848-2cd1-4c98-913e-f567e6d02f1a-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"f31a1848-2cd1-4c98-913e-f567e6d02f1a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:17.482597 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.482577 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7216f712-5ed6-456d-bf6b-40c326ce28aa" path="/var/lib/kubelet/pods/7216f712-5ed6-456d-bf6b-40c326ce28aa/volumes" Oct 31 00:37:17.582475 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.582447 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84jvr\" (UniqueName: \"kubernetes.io/projected/f31a1848-2cd1-4c98-913e-f567e6d02f1a-kube-api-access-84jvr\") pod \"data-science-collector-collector-0\" (UID: \"f31a1848-2cd1-4c98-913e-f567e6d02f1a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:17.582613 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.582490 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f31a1848-2cd1-4c98-913e-f567e6d02f1a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"f31a1848-2cd1-4c98-913e-f567e6d02f1a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:17.582613 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.582541 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f31a1848-2cd1-4c98-913e-f567e6d02f1a-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"f31a1848-2cd1-4c98-913e-f567e6d02f1a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:17.582613 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.582576 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f31a1848-2cd1-4c98-913e-f567e6d02f1a-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"f31a1848-2cd1-4c98-913e-f567e6d02f1a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:17.582785 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.582611 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f31a1848-2cd1-4c98-913e-f567e6d02f1a-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"f31a1848-2cd1-4c98-913e-f567e6d02f1a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:17.582785 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.582662 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f31a1848-2cd1-4c98-913e-f567e6d02f1a-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"f31a1848-2cd1-4c98-913e-f567e6d02f1a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:17.583427 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.583404 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f31a1848-2cd1-4c98-913e-f567e6d02f1a-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"f31a1848-2cd1-4c98-913e-f567e6d02f1a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:17.585127 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.585103 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f31a1848-2cd1-4c98-913e-f567e6d02f1a-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"f31a1848-2cd1-4c98-913e-f567e6d02f1a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:17.585301 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.585278 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f31a1848-2cd1-4c98-913e-f567e6d02f1a-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"f31a1848-2cd1-4c98-913e-f567e6d02f1a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:17.585442 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.585427 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f31a1848-2cd1-4c98-913e-f567e6d02f1a-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"f31a1848-2cd1-4c98-913e-f567e6d02f1a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:17.585508 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.585490 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f31a1848-2cd1-4c98-913e-f567e6d02f1a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"f31a1848-2cd1-4c98-913e-f567e6d02f1a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:17.594656 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.594609 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84jvr\" (UniqueName: \"kubernetes.io/projected/f31a1848-2cd1-4c98-913e-f567e6d02f1a-kube-api-access-84jvr\") pod \"data-science-collector-collector-0\" (UID: \"f31a1848-2cd1-4c98-913e-f567e6d02f1a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:17.706048 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.706025 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:17.980799 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.980775 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:37:17.983028 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:37:17.983002 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf31a1848_2cd1_4c98_913e_f567e6d02f1a.slice/crio-bf2286e816c13fdc2a63f53cf274ee90248c51fa4b87de280f2473d3886ee8ba WatchSource:0}: Error finding container bf2286e816c13fdc2a63f53cf274ee90248c51fa4b87de280f2473d3886ee8ba: Status 404 returned error can't find the container with id bf2286e816c13fdc2a63f53cf274ee90248c51fa4b87de280f2473d3886ee8ba Oct 31 00:37:17.988183 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.988146 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:37:17.988290 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:17.988212 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:37:18.037240 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:18.037212 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"f31a1848-2cd1-4c98-913e-f567e6d02f1a","Type":"ContainerStarted","Data":"bf2286e816c13fdc2a63f53cf274ee90248c51fa4b87de280f2473d3886ee8ba"} Oct 31 00:37:19.044104 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:19.044070 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"f31a1848-2cd1-4c98-913e-f567e6d02f1a","Type":"ContainerStarted","Data":"8b2bf4912beb7307ee06d1e3c82bc51a3b3766ad690cb4c1c2697601779164de"} Oct 31 00:37:19.065489 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:19.065435 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.065421529 podStartE2EDuration="2.065421529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:19.063634929 +0000 UTC m=+2275.700453647" watchObservedRunningTime="2026-10-31 00:37:19.065421529 +0000 UTC m=+2275.702240244" Oct 31 00:37:20.899637 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:37:20.899595 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:37:22.007087 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:22.007062 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-r2sfh" Oct 31 00:37:22.053980 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:22.053931 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-wdnls"] Oct 31 00:37:22.054888 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:22.054840 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wdnls" podUID="72eedfa0-5a3d-4047-98ea-c5e9f5582552" containerName="korrel8r" containerID="cri-o://e599d01e41275c0d3668936435933279f22cd23e028ba14707e3e1f590a01ac3" gracePeriod=30 Oct 31 00:37:22.219537 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:22.219509 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-gcwfl"] Oct 31 00:37:22.228289 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:22.228257 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gcwfl" Oct 31 00:37:22.234582 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:22.234557 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-gcwfl"] Oct 31 00:37:22.316167 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:22.316143 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wdnls" Oct 31 00:37:22.325179 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:22.325156 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzc4t\" (UniqueName: \"kubernetes.io/projected/53262348-aaeb-4648-8b93-3b4c40f3ebc2-kube-api-access-gzc4t\") pod \"korrel8r-5cbcc89b8f-gcwfl\" (UID: \"53262348-aaeb-4648-8b93-3b4c40f3ebc2\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gcwfl" Oct 31 00:37:22.325292 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:22.325225 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/53262348-aaeb-4648-8b93-3b4c40f3ebc2-config\") pod \"korrel8r-5cbcc89b8f-gcwfl\" (UID: \"53262348-aaeb-4648-8b93-3b4c40f3ebc2\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gcwfl" Oct 31 00:37:22.325334 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:22.325310 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/53262348-aaeb-4648-8b93-3b4c40f3ebc2-tls\") pod \"korrel8r-5cbcc89b8f-gcwfl\" (UID: \"53262348-aaeb-4648-8b93-3b4c40f3ebc2\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gcwfl" Oct 31 00:37:22.426246 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:22.426212 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72eedfa0-5a3d-4047-98ea-c5e9f5582552-config\" (UniqueName: \"kubernetes.io/configmap/72eedfa0-5a3d-4047-98ea-c5e9f5582552-config\") pod \"72eedfa0-5a3d-4047-98ea-c5e9f5582552\" (UID: \"72eedfa0-5a3d-4047-98ea-c5e9f5582552\") " Oct 31 00:37:22.426408 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:22.426269 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72eedfa0-5a3d-4047-98ea-c5e9f5582552-tls\" (UniqueName: \"kubernetes.io/secret/72eedfa0-5a3d-4047-98ea-c5e9f5582552-tls\") pod \"72eedfa0-5a3d-4047-98ea-c5e9f5582552\" (UID: \"72eedfa0-5a3d-4047-98ea-c5e9f5582552\") " Oct 31 00:37:22.426408 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:22.426311 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72eedfa0-5a3d-4047-98ea-c5e9f5582552-kube-api-access-nk2sw\" (UniqueName: \"kubernetes.io/projected/72eedfa0-5a3d-4047-98ea-c5e9f5582552-kube-api-access-nk2sw\") pod \"72eedfa0-5a3d-4047-98ea-c5e9f5582552\" (UID: \"72eedfa0-5a3d-4047-98ea-c5e9f5582552\") " Oct 31 00:37:22.426561 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:22.426540 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/53262348-aaeb-4648-8b93-3b4c40f3ebc2-config\") pod \"korrel8r-5cbcc89b8f-gcwfl\" (UID: \"53262348-aaeb-4648-8b93-3b4c40f3ebc2\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gcwfl" Oct 31 00:37:22.426621 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:22.426590 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/53262348-aaeb-4648-8b93-3b4c40f3ebc2-tls\") pod \"korrel8r-5cbcc89b8f-gcwfl\" (UID: \"53262348-aaeb-4648-8b93-3b4c40f3ebc2\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gcwfl" Oct 31 00:37:22.426683 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:22.426657 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzc4t\" (UniqueName: \"kubernetes.io/projected/53262348-aaeb-4648-8b93-3b4c40f3ebc2-kube-api-access-gzc4t\") pod \"korrel8r-5cbcc89b8f-gcwfl\" (UID: \"53262348-aaeb-4648-8b93-3b4c40f3ebc2\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gcwfl" Oct 31 00:37:22.426738 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:22.426657 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72eedfa0-5a3d-4047-98ea-c5e9f5582552-config" pod "72eedfa0-5a3d-4047-98ea-c5e9f5582552" (UID: "72eedfa0-5a3d-4047-98ea-c5e9f5582552"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:22.427404 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:22.427373 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/53262348-aaeb-4648-8b93-3b4c40f3ebc2-config\") pod \"korrel8r-5cbcc89b8f-gcwfl\" (UID: \"53262348-aaeb-4648-8b93-3b4c40f3ebc2\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gcwfl" Oct 31 00:37:22.428441 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:22.428412 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72eedfa0-5a3d-4047-98ea-c5e9f5582552-kube-api-access-nk2sw" pod "72eedfa0-5a3d-4047-98ea-c5e9f5582552" (UID: "72eedfa0-5a3d-4047-98ea-c5e9f5582552"). InnerVolumeSpecName "kube-api-access-nk2sw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:22.428541 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:22.428422 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72eedfa0-5a3d-4047-98ea-c5e9f5582552-tls" pod "72eedfa0-5a3d-4047-98ea-c5e9f5582552" (UID: "72eedfa0-5a3d-4047-98ea-c5e9f5582552"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:22.429070 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:22.429050 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/53262348-aaeb-4648-8b93-3b4c40f3ebc2-tls\") pod \"korrel8r-5cbcc89b8f-gcwfl\" (UID: \"53262348-aaeb-4648-8b93-3b4c40f3ebc2\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gcwfl" Oct 31 00:37:22.438634 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:22.438615 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzc4t\" (UniqueName: \"kubernetes.io/projected/53262348-aaeb-4648-8b93-3b4c40f3ebc2-kube-api-access-gzc4t\") pod \"korrel8r-5cbcc89b8f-gcwfl\" (UID: \"53262348-aaeb-4648-8b93-3b4c40f3ebc2\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gcwfl" Oct 31 00:37:22.527609 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:22.527588 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/72eedfa0-5a3d-4047-98ea-c5e9f5582552-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:22.527609 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:22.527607 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/72eedfa0-5a3d-4047-98ea-c5e9f5582552-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:22.527733 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:22.527617 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk2sw\" (UniqueName: \"kubernetes.io/projected/72eedfa0-5a3d-4047-98ea-c5e9f5582552-kube-api-access-nk2sw\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:22.545672 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:22.545613 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gcwfl" Oct 31 00:37:22.826590 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:22.826564 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-gcwfl"] Oct 31 00:37:22.828205 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:37:22.828169 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53262348_aaeb_4648_8b93_3b4c40f3ebc2.slice/crio-3aa9ba7edbeb8c5cf819c502637ecacbffbc7d15b7e38a394f58baaad2ecc5a6 WatchSource:0}: Error finding container 3aa9ba7edbeb8c5cf819c502637ecacbffbc7d15b7e38a394f58baaad2ecc5a6: Status 404 returned error can't find the container with id 3aa9ba7edbeb8c5cf819c502637ecacbffbc7d15b7e38a394f58baaad2ecc5a6 Oct 31 00:37:23.066850 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:23.066820 5274 generic.go:356] "Generic (PLEG): container finished" podID="72eedfa0-5a3d-4047-98ea-c5e9f5582552" containerID="e599d01e41275c0d3668936435933279f22cd23e028ba14707e3e1f590a01ac3" exitCode=0 Oct 31 00:37:23.067309 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:23.066870 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wdnls" Oct 31 00:37:23.067309 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:23.066913 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wdnls" event={"ID":"72eedfa0-5a3d-4047-98ea-c5e9f5582552","Type":"ContainerDied","Data":"e599d01e41275c0d3668936435933279f22cd23e028ba14707e3e1f590a01ac3"} Oct 31 00:37:23.067309 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:23.066977 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-wdnls" event={"ID":"72eedfa0-5a3d-4047-98ea-c5e9f5582552","Type":"ContainerDied","Data":"5bac39bff688ae2382dcd7423982cd1ffa527758125be94cc4ff7e2250a05855"} Oct 31 00:37:23.067309 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:23.067008 5274 scope.go:122] "RemoveContainer" containerID="e599d01e41275c0d3668936435933279f22cd23e028ba14707e3e1f590a01ac3" Oct 31 00:37:23.069027 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:23.069002 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gcwfl" event={"ID":"53262348-aaeb-4648-8b93-3b4c40f3ebc2","Type":"ContainerStarted","Data":"7927106a65370a34199843d6da7a61c19272426c8d67f88f57e0db714d50a0ac"} Oct 31 00:37:23.069148 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:23.069050 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gcwfl" event={"ID":"53262348-aaeb-4648-8b93-3b4c40f3ebc2","Type":"ContainerStarted","Data":"3aa9ba7edbeb8c5cf819c502637ecacbffbc7d15b7e38a394f58baaad2ecc5a6"} Oct 31 00:37:23.069148 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:23.069080 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gcwfl" Oct 31 00:37:23.079715 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:23.079699 5274 scope.go:122] "RemoveContainer" containerID="e599d01e41275c0d3668936435933279f22cd23e028ba14707e3e1f590a01ac3" Oct 31 00:37:23.079990 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:37:23.079969 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e599d01e41275c0d3668936435933279f22cd23e028ba14707e3e1f590a01ac3\": container with ID starting with e599d01e41275c0d3668936435933279f22cd23e028ba14707e3e1f590a01ac3 not found: ID does not exist" containerID="e599d01e41275c0d3668936435933279f22cd23e028ba14707e3e1f590a01ac3" Oct 31 00:37:23.080060 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:23.080003 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e599d01e41275c0d3668936435933279f22cd23e028ba14707e3e1f590a01ac3"} err="failed to get container status \"e599d01e41275c0d3668936435933279f22cd23e028ba14707e3e1f590a01ac3\": rpc error: code = NotFound desc = could not find container \"e599d01e41275c0d3668936435933279f22cd23e028ba14707e3e1f590a01ac3\": container with ID starting with e599d01e41275c0d3668936435933279f22cd23e028ba14707e3e1f590a01ac3 not found: ID does not exist" Oct 31 00:37:23.089594 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:23.089556 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gcwfl" podStartSLOduration=1.089544901 podStartE2EDuration="1.089544901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:23.088463448 +0000 UTC m=+2279.725282166" watchObservedRunningTime="2026-10-31 00:37:23.089544901 +0000 UTC m=+2279.726363659" Oct 31 00:37:23.102600 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:23.102574 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-wdnls"] Oct 31 00:37:23.105136 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:23.105116 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-wdnls"] Oct 31 00:37:23.480856 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:23.480827 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72eedfa0-5a3d-4047-98ea-c5e9f5582552" path="/var/lib/kubelet/pods/72eedfa0-5a3d-4047-98ea-c5e9f5582552/volumes" Oct 31 00:37:24.698288 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:24.698257 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:37:24.699175 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:24.698449 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="f31a1848-2cd1-4c98-913e-f567e6d02f1a" containerName="otc-container" containerID="cri-o://8b2bf4912beb7307ee06d1e3c82bc51a3b3766ad690cb4c1c2697601779164de" gracePeriod=30 Oct 31 00:37:24.943265 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:24.943245 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:25.047351 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.047274 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f31a1848-2cd1-4c98-913e-f567e6d02f1a-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f31a1848-2cd1-4c98-913e-f567e6d02f1a-otlp-tls-certs\") pod \"f31a1848-2cd1-4c98-913e-f567e6d02f1a\" (UID: \"f31a1848-2cd1-4c98-913e-f567e6d02f1a\") " Oct 31 00:37:25.047351 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.047326 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f31a1848-2cd1-4c98-913e-f567e6d02f1a-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f31a1848-2cd1-4c98-913e-f567e6d02f1a-monitoring-tls-certs\") pod \"f31a1848-2cd1-4c98-913e-f567e6d02f1a\" (UID: \"f31a1848-2cd1-4c98-913e-f567e6d02f1a\") " Oct 31 00:37:25.047579 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.047355 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f31a1848-2cd1-4c98-913e-f567e6d02f1a-tls-certs\" (UniqueName: \"kubernetes.io/secret/f31a1848-2cd1-4c98-913e-f567e6d02f1a-tls-certs\") pod \"f31a1848-2cd1-4c98-913e-f567e6d02f1a\" (UID: \"f31a1848-2cd1-4c98-913e-f567e6d02f1a\") " Oct 31 00:37:25.047579 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.047410 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f31a1848-2cd1-4c98-913e-f567e6d02f1a-otc-internal\" (UniqueName: \"kubernetes.io/configmap/f31a1848-2cd1-4c98-913e-f567e6d02f1a-otc-internal\") pod \"f31a1848-2cd1-4c98-913e-f567e6d02f1a\" (UID: \"f31a1848-2cd1-4c98-913e-f567e6d02f1a\") " Oct 31 00:37:25.047579 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.047447 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f31a1848-2cd1-4c98-913e-f567e6d02f1a-kube-api-access-84jvr\" (UniqueName: \"kubernetes.io/projected/f31a1848-2cd1-4c98-913e-f567e6d02f1a-kube-api-access-84jvr\") pod \"f31a1848-2cd1-4c98-913e-f567e6d02f1a\" (UID: \"f31a1848-2cd1-4c98-913e-f567e6d02f1a\") " Oct 31 00:37:25.047579 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.047528 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f31a1848-2cd1-4c98-913e-f567e6d02f1a-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f31a1848-2cd1-4c98-913e-f567e6d02f1a-data-science-collector-ta-client-cert\") pod \"f31a1848-2cd1-4c98-913e-f567e6d02f1a\" (UID: \"f31a1848-2cd1-4c98-913e-f567e6d02f1a\") " Oct 31 00:37:25.048130 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.048095 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f31a1848-2cd1-4c98-913e-f567e6d02f1a-otc-internal" pod "f31a1848-2cd1-4c98-913e-f567e6d02f1a" (UID: "f31a1848-2cd1-4c98-913e-f567e6d02f1a"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:25.049612 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.049582 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f31a1848-2cd1-4c98-913e-f567e6d02f1a-kube-api-access-84jvr" pod "f31a1848-2cd1-4c98-913e-f567e6d02f1a" (UID: "f31a1848-2cd1-4c98-913e-f567e6d02f1a"). InnerVolumeSpecName "kube-api-access-84jvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:25.049612 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.049598 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f31a1848-2cd1-4c98-913e-f567e6d02f1a-tls-certs" pod "f31a1848-2cd1-4c98-913e-f567e6d02f1a" (UID: "f31a1848-2cd1-4c98-913e-f567e6d02f1a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:25.049779 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.049689 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f31a1848-2cd1-4c98-913e-f567e6d02f1a-otlp-tls-certs" pod "f31a1848-2cd1-4c98-913e-f567e6d02f1a" (UID: "f31a1848-2cd1-4c98-913e-f567e6d02f1a"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:25.049868 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.049853 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f31a1848-2cd1-4c98-913e-f567e6d02f1a-monitoring-tls-certs" pod "f31a1848-2cd1-4c98-913e-f567e6d02f1a" (UID: "f31a1848-2cd1-4c98-913e-f567e6d02f1a"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:25.049990 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.049950 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f31a1848-2cd1-4c98-913e-f567e6d02f1a-data-science-collector-ta-client-cert" pod "f31a1848-2cd1-4c98-913e-f567e6d02f1a" (UID: "f31a1848-2cd1-4c98-913e-f567e6d02f1a"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:25.080693 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.080665 5274 generic.go:356] "Generic (PLEG): container finished" podID="f31a1848-2cd1-4c98-913e-f567e6d02f1a" containerID="8b2bf4912beb7307ee06d1e3c82bc51a3b3766ad690cb4c1c2697601779164de" exitCode=0 Oct 31 00:37:25.080813 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.080718 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:25.080813 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.080748 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"f31a1848-2cd1-4c98-913e-f567e6d02f1a","Type":"ContainerDied","Data":"8b2bf4912beb7307ee06d1e3c82bc51a3b3766ad690cb4c1c2697601779164de"} Oct 31 00:37:25.080813 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.080804 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"f31a1848-2cd1-4c98-913e-f567e6d02f1a","Type":"ContainerDied","Data":"bf2286e816c13fdc2a63f53cf274ee90248c51fa4b87de280f2473d3886ee8ba"} Oct 31 00:37:25.080979 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.080826 5274 scope.go:122] "RemoveContainer" containerID="8b2bf4912beb7307ee06d1e3c82bc51a3b3766ad690cb4c1c2697601779164de" Oct 31 00:37:25.090492 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.090475 5274 scope.go:122] "RemoveContainer" containerID="8b2bf4912beb7307ee06d1e3c82bc51a3b3766ad690cb4c1c2697601779164de" Oct 31 00:37:25.090748 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:37:25.090731 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b2bf4912beb7307ee06d1e3c82bc51a3b3766ad690cb4c1c2697601779164de\": container with ID starting with 8b2bf4912beb7307ee06d1e3c82bc51a3b3766ad690cb4c1c2697601779164de not found: ID does not exist" containerID="8b2bf4912beb7307ee06d1e3c82bc51a3b3766ad690cb4c1c2697601779164de" Oct 31 00:37:25.090794 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.090756 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b2bf4912beb7307ee06d1e3c82bc51a3b3766ad690cb4c1c2697601779164de"} err="failed to get container status \"8b2bf4912beb7307ee06d1e3c82bc51a3b3766ad690cb4c1c2697601779164de\": rpc error: code = NotFound desc = could not find container \"8b2bf4912beb7307ee06d1e3c82bc51a3b3766ad690cb4c1c2697601779164de\": container with ID starting with 8b2bf4912beb7307ee06d1e3c82bc51a3b3766ad690cb4c1c2697601779164de not found: ID does not exist" Oct 31 00:37:25.103942 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.103916 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:37:25.107868 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.107847 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:37:25.133673 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.133653 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:37:25.134037 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.134024 5274 cpu_manager.go:414] "containerMap: removing container" podUID="72eedfa0-5a3d-4047-98ea-c5e9f5582552" containerName="korrel8r" Oct 31 00:37:25.134090 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.134039 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72eedfa0-5a3d-4047-98ea-c5e9f5582552" containerName="korrel8r" Oct 31 00:37:25.134090 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.134072 5274 cpu_manager.go:414] "containerMap: removing container" podUID="f31a1848-2cd1-4c98-913e-f567e6d02f1a" containerName="otc-container" Oct 31 00:37:25.134090 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.134079 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f31a1848-2cd1-4c98-913e-f567e6d02f1a" containerName="otc-container" Oct 31 00:37:25.134236 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.134149 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="f31a1848-2cd1-4c98-913e-f567e6d02f1a" containerName="otc-container" Oct 31 00:37:25.134236 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.134164 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="72eedfa0-5a3d-4047-98ea-c5e9f5582552" containerName="korrel8r" Oct 31 00:37:25.138375 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.138358 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:25.151148 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.149890 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:37:25.151148 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.149978 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:37:25.151148 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.150002 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:37:25.151148 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.150493 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-cfb2n\"" Oct 31 00:37:25.151148 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.150623 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:37:25.151148 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.150925 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:37:25.153567 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.153268 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/def1055e-9b8b-4ad6-9203-b4f76835ec10-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"def1055e-9b8b-4ad6-9203-b4f76835ec10\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:25.153567 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.153329 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/def1055e-9b8b-4ad6-9203-b4f76835ec10-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"def1055e-9b8b-4ad6-9203-b4f76835ec10\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:25.153567 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.153357 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/def1055e-9b8b-4ad6-9203-b4f76835ec10-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"def1055e-9b8b-4ad6-9203-b4f76835ec10\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:25.153766 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.153602 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def1055e-9b8b-4ad6-9203-b4f76835ec10-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"def1055e-9b8b-4ad6-9203-b4f76835ec10\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:25.155576 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.155551 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:37:25.157370 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.157345 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/def1055e-9b8b-4ad6-9203-b4f76835ec10-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"def1055e-9b8b-4ad6-9203-b4f76835ec10\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:25.157511 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.157492 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6z9l\" (UniqueName: \"kubernetes.io/projected/def1055e-9b8b-4ad6-9203-b4f76835ec10-kube-api-access-d6z9l\") pod \"data-science-collector-collector-0\" (UID: \"def1055e-9b8b-4ad6-9203-b4f76835ec10\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:25.157625 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.157594 5274 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f31a1848-2cd1-4c98-913e-f567e6d02f1a-monitoring-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:25.157625 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.157616 5274 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f31a1848-2cd1-4c98-913e-f567e6d02f1a-otc-internal\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:25.157807 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.157631 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84jvr\" (UniqueName: \"kubernetes.io/projected/f31a1848-2cd1-4c98-913e-f567e6d02f1a-kube-api-access-84jvr\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:25.157807 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.157643 5274 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f31a1848-2cd1-4c98-913e-f567e6d02f1a-otlp-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:25.157807 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.157654 5274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f31a1848-2cd1-4c98-913e-f567e6d02f1a-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:25.157807 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.157664 5274 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f31a1848-2cd1-4c98-913e-f567e6d02f1a-data-science-collector-ta-client-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:25.258093 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.258060 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/def1055e-9b8b-4ad6-9203-b4f76835ec10-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"def1055e-9b8b-4ad6-9203-b4f76835ec10\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:25.258255 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.258106 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/def1055e-9b8b-4ad6-9203-b4f76835ec10-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"def1055e-9b8b-4ad6-9203-b4f76835ec10\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:25.258255 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.258124 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/def1055e-9b8b-4ad6-9203-b4f76835ec10-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"def1055e-9b8b-4ad6-9203-b4f76835ec10\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:25.258255 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.258148 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def1055e-9b8b-4ad6-9203-b4f76835ec10-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"def1055e-9b8b-4ad6-9203-b4f76835ec10\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:25.258255 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.258164 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/def1055e-9b8b-4ad6-9203-b4f76835ec10-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"def1055e-9b8b-4ad6-9203-b4f76835ec10\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:25.258255 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.258209 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6z9l\" (UniqueName: \"kubernetes.io/projected/def1055e-9b8b-4ad6-9203-b4f76835ec10-kube-api-access-d6z9l\") pod \"data-science-collector-collector-0\" (UID: \"def1055e-9b8b-4ad6-9203-b4f76835ec10\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:25.258899 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.258859 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/def1055e-9b8b-4ad6-9203-b4f76835ec10-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"def1055e-9b8b-4ad6-9203-b4f76835ec10\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:25.260682 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.260658 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/def1055e-9b8b-4ad6-9203-b4f76835ec10-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"def1055e-9b8b-4ad6-9203-b4f76835ec10\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:25.260682 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.260678 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/def1055e-9b8b-4ad6-9203-b4f76835ec10-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"def1055e-9b8b-4ad6-9203-b4f76835ec10\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:25.261011 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.260994 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def1055e-9b8b-4ad6-9203-b4f76835ec10-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"def1055e-9b8b-4ad6-9203-b4f76835ec10\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:25.261062 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.261048 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/def1055e-9b8b-4ad6-9203-b4f76835ec10-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"def1055e-9b8b-4ad6-9203-b4f76835ec10\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:25.270093 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.270075 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6z9l\" (UniqueName: \"kubernetes.io/projected/def1055e-9b8b-4ad6-9203-b4f76835ec10-kube-api-access-d6z9l\") pod \"data-science-collector-collector-0\" (UID: \"def1055e-9b8b-4ad6-9203-b4f76835ec10\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:25.459038 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.459013 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:25.481005 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.480973 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f31a1848-2cd1-4c98-913e-f567e6d02f1a" path="/var/lib/kubelet/pods/f31a1848-2cd1-4c98-913e-f567e6d02f1a/volumes" Oct 31 00:37:25.733781 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.733755 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:37:25.738857 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:37:25.738827 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddef1055e_9b8b_4ad6_9203_b4f76835ec10.slice/crio-97da7d0c3f0ff43243b420133020ddb9d5e8368fa824a3e44270b91c194d7a3c WatchSource:0}: Error finding container 97da7d0c3f0ff43243b420133020ddb9d5e8368fa824a3e44270b91c194d7a3c: Status 404 returned error can't find the container with id 97da7d0c3f0ff43243b420133020ddb9d5e8368fa824a3e44270b91c194d7a3c Oct 31 00:37:25.741937 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.741901 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:37:25.742032 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:25.741998 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:37:26.087274 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:26.087186 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"def1055e-9b8b-4ad6-9203-b4f76835ec10","Type":"ContainerStarted","Data":"0237e7ff3200633864d2a182d48768e7e5079c871fb22f2fdec8c0d2c0111367"} Oct 31 00:37:26.087274 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:26.087250 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"def1055e-9b8b-4ad6-9203-b4f76835ec10","Type":"ContainerStarted","Data":"97da7d0c3f0ff43243b420133020ddb9d5e8368fa824a3e44270b91c194d7a3c"} Oct 31 00:37:26.108298 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:26.108234 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.108220988 podStartE2EDuration="1.108220988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:26.107486936 +0000 UTC m=+2282.744305653" watchObservedRunningTime="2026-10-31 00:37:26.108220988 +0000 UTC m=+2282.745039703" Oct 31 00:37:34.075634 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:34.075598 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gcwfl" Oct 31 00:37:34.121797 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:34.121767 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-r2sfh"] Oct 31 00:37:34.122072 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:34.122012 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-r2sfh" podUID="9f61d0fb-44a1-4c82-a9ca-ef596732f2ba" containerName="korrel8r" containerID="cri-o://25a1e71fea134e8d15519976ba3a3460f472625be38b7637316a542d666753dd" gracePeriod=30 Oct 31 00:37:34.365741 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:34.365716 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-r2sfh" Oct 31 00:37:34.437915 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:34.437887 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f61d0fb-44a1-4c82-a9ca-ef596732f2ba-config\" (UniqueName: \"kubernetes.io/configmap/9f61d0fb-44a1-4c82-a9ca-ef596732f2ba-config\") pod \"9f61d0fb-44a1-4c82-a9ca-ef596732f2ba\" (UID: \"9f61d0fb-44a1-4c82-a9ca-ef596732f2ba\") " Oct 31 00:37:34.438093 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:34.437977 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f61d0fb-44a1-4c82-a9ca-ef596732f2ba-tls\" (UniqueName: \"kubernetes.io/secret/9f61d0fb-44a1-4c82-a9ca-ef596732f2ba-tls\") pod \"9f61d0fb-44a1-4c82-a9ca-ef596732f2ba\" (UID: \"9f61d0fb-44a1-4c82-a9ca-ef596732f2ba\") " Oct 31 00:37:34.438093 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:34.438014 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f61d0fb-44a1-4c82-a9ca-ef596732f2ba-kube-api-access-2n6xr\" (UniqueName: \"kubernetes.io/projected/9f61d0fb-44a1-4c82-a9ca-ef596732f2ba-kube-api-access-2n6xr\") pod \"9f61d0fb-44a1-4c82-a9ca-ef596732f2ba\" (UID: \"9f61d0fb-44a1-4c82-a9ca-ef596732f2ba\") " Oct 31 00:37:34.438367 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:34.438334 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f61d0fb-44a1-4c82-a9ca-ef596732f2ba-config" pod "9f61d0fb-44a1-4c82-a9ca-ef596732f2ba" (UID: "9f61d0fb-44a1-4c82-a9ca-ef596732f2ba"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:34.440123 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:34.440096 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f61d0fb-44a1-4c82-a9ca-ef596732f2ba-kube-api-access-2n6xr" pod "9f61d0fb-44a1-4c82-a9ca-ef596732f2ba" (UID: "9f61d0fb-44a1-4c82-a9ca-ef596732f2ba"). InnerVolumeSpecName "kube-api-access-2n6xr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:34.440201 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:34.440161 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f61d0fb-44a1-4c82-a9ca-ef596732f2ba-tls" pod "9f61d0fb-44a1-4c82-a9ca-ef596732f2ba" (UID: "9f61d0fb-44a1-4c82-a9ca-ef596732f2ba"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:34.538926 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:34.538900 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9f61d0fb-44a1-4c82-a9ca-ef596732f2ba-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:34.538926 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:34.538926 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/9f61d0fb-44a1-4c82-a9ca-ef596732f2ba-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:34.539083 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:34.538940 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n6xr\" (UniqueName: \"kubernetes.io/projected/9f61d0fb-44a1-4c82-a9ca-ef596732f2ba-kube-api-access-2n6xr\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:34.836069 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:34.836038 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-gcj9x"] Oct 31 00:37:34.836419 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:34.836406 5274 cpu_manager.go:414] "containerMap: removing container" podUID="9f61d0fb-44a1-4c82-a9ca-ef596732f2ba" containerName="korrel8r" Oct 31 00:37:34.836465 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:34.836421 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f61d0fb-44a1-4c82-a9ca-ef596732f2ba" containerName="korrel8r" Oct 31 00:37:34.836506 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:34.836497 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f61d0fb-44a1-4c82-a9ca-ef596732f2ba" containerName="korrel8r" Oct 31 00:37:34.839722 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:34.839707 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-gcj9x" Oct 31 00:37:34.848263 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:34.848240 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-gcj9x"] Oct 31 00:37:34.941893 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:34.941861 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/7f0518ab-c116-4b78-9079-9f45d67d1f1a-tls\") pod \"korrel8r-66d6f7f696-gcj9x\" (UID: \"7f0518ab-c116-4b78-9079-9f45d67d1f1a\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-gcj9x" Oct 31 00:37:34.942067 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:34.941946 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7f0518ab-c116-4b78-9079-9f45d67d1f1a-config\") pod \"korrel8r-66d6f7f696-gcj9x\" (UID: \"7f0518ab-c116-4b78-9079-9f45d67d1f1a\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-gcj9x" Oct 31 00:37:34.942114 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:34.942060 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp8xj\" (UniqueName: \"kubernetes.io/projected/7f0518ab-c116-4b78-9079-9f45d67d1f1a-kube-api-access-zp8xj\") pod \"korrel8r-66d6f7f696-gcj9x\" (UID: \"7f0518ab-c116-4b78-9079-9f45d67d1f1a\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-gcj9x" Oct 31 00:37:35.042787 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:35.042751 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp8xj\" (UniqueName: \"kubernetes.io/projected/7f0518ab-c116-4b78-9079-9f45d67d1f1a-kube-api-access-zp8xj\") pod \"korrel8r-66d6f7f696-gcj9x\" (UID: \"7f0518ab-c116-4b78-9079-9f45d67d1f1a\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-gcj9x" Oct 31 00:37:35.043025 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:35.042814 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/7f0518ab-c116-4b78-9079-9f45d67d1f1a-tls\") pod \"korrel8r-66d6f7f696-gcj9x\" (UID: \"7f0518ab-c116-4b78-9079-9f45d67d1f1a\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-gcj9x" Oct 31 00:37:35.043025 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:35.042838 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7f0518ab-c116-4b78-9079-9f45d67d1f1a-config\") pod \"korrel8r-66d6f7f696-gcj9x\" (UID: \"7f0518ab-c116-4b78-9079-9f45d67d1f1a\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-gcj9x" Oct 31 00:37:35.043645 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:35.043627 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7f0518ab-c116-4b78-9079-9f45d67d1f1a-config\") pod \"korrel8r-66d6f7f696-gcj9x\" (UID: \"7f0518ab-c116-4b78-9079-9f45d67d1f1a\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-gcj9x" Oct 31 00:37:35.045438 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:35.045416 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/7f0518ab-c116-4b78-9079-9f45d67d1f1a-tls\") pod \"korrel8r-66d6f7f696-gcj9x\" (UID: \"7f0518ab-c116-4b78-9079-9f45d67d1f1a\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-gcj9x" Oct 31 00:37:35.055238 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:35.055217 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp8xj\" (UniqueName: \"kubernetes.io/projected/7f0518ab-c116-4b78-9079-9f45d67d1f1a-kube-api-access-zp8xj\") pod \"korrel8r-66d6f7f696-gcj9x\" (UID: \"7f0518ab-c116-4b78-9079-9f45d67d1f1a\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-gcj9x" Oct 31 00:37:35.139481 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:35.139447 5274 generic.go:356] "Generic (PLEG): container finished" podID="9f61d0fb-44a1-4c82-a9ca-ef596732f2ba" containerID="25a1e71fea134e8d15519976ba3a3460f472625be38b7637316a542d666753dd" exitCode=0 Oct 31 00:37:35.139882 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:35.139500 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-r2sfh" Oct 31 00:37:35.139882 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:35.139533 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-r2sfh" event={"ID":"9f61d0fb-44a1-4c82-a9ca-ef596732f2ba","Type":"ContainerDied","Data":"25a1e71fea134e8d15519976ba3a3460f472625be38b7637316a542d666753dd"} Oct 31 00:37:35.139882 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:35.139587 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-r2sfh" event={"ID":"9f61d0fb-44a1-4c82-a9ca-ef596732f2ba","Type":"ContainerDied","Data":"f697734b18e89afe74b4709263efdcdc0eda7858e79da48dfefe43ea99cf8600"} Oct 31 00:37:35.139882 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:35.139615 5274 scope.go:122] "RemoveContainer" containerID="25a1e71fea134e8d15519976ba3a3460f472625be38b7637316a542d666753dd" Oct 31 00:37:35.150516 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:35.150494 5274 scope.go:122] "RemoveContainer" containerID="25a1e71fea134e8d15519976ba3a3460f472625be38b7637316a542d666753dd" Oct 31 00:37:35.150783 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:37:35.150761 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25a1e71fea134e8d15519976ba3a3460f472625be38b7637316a542d666753dd\": container with ID starting with 25a1e71fea134e8d15519976ba3a3460f472625be38b7637316a542d666753dd not found: ID does not exist" containerID="25a1e71fea134e8d15519976ba3a3460f472625be38b7637316a542d666753dd" Oct 31 00:37:35.150859 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:35.150795 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25a1e71fea134e8d15519976ba3a3460f472625be38b7637316a542d666753dd"} err="failed to get container status \"25a1e71fea134e8d15519976ba3a3460f472625be38b7637316a542d666753dd\": rpc error: code = NotFound desc = could not find container \"25a1e71fea134e8d15519976ba3a3460f472625be38b7637316a542d666753dd\": container with ID starting with 25a1e71fea134e8d15519976ba3a3460f472625be38b7637316a542d666753dd not found: ID does not exist" Oct 31 00:37:35.153962 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:35.153926 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-gcj9x" Oct 31 00:37:35.168362 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:35.168340 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-r2sfh"] Oct 31 00:37:35.174647 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:35.174619 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-r2sfh"] Oct 31 00:37:35.429037 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:35.428939 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-gcj9x"] Oct 31 00:37:35.431483 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:37:35.431451 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f0518ab_c116_4b78_9079_9f45d67d1f1a.slice/crio-415090a30b6afefa24857f0ea423958ccd288b7ae760e29a4549a86c4555b78e WatchSource:0}: Error finding container 415090a30b6afefa24857f0ea423958ccd288b7ae760e29a4549a86c4555b78e: Status 404 returned error can't find the container with id 415090a30b6afefa24857f0ea423958ccd288b7ae760e29a4549a86c4555b78e Oct 31 00:37:35.482456 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:35.482429 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f61d0fb-44a1-4c82-a9ca-ef596732f2ba" path="/var/lib/kubelet/pods/9f61d0fb-44a1-4c82-a9ca-ef596732f2ba/volumes" Oct 31 00:37:36.146364 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:36.146328 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-gcj9x" event={"ID":"7f0518ab-c116-4b78-9079-9f45d67d1f1a","Type":"ContainerStarted","Data":"03a41eed5f69cfee720cac0f538cc5d029646fed4ec8fd63a4f41975c6614d48"} Oct 31 00:37:36.146757 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:36.146381 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-gcj9x" event={"ID":"7f0518ab-c116-4b78-9079-9f45d67d1f1a","Type":"ContainerStarted","Data":"415090a30b6afefa24857f0ea423958ccd288b7ae760e29a4549a86c4555b78e"} Oct 31 00:37:36.146757 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:36.146447 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-gcj9x" Oct 31 00:37:36.172353 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:36.172293 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-gcj9x" podStartSLOduration=2.172272205 podStartE2EDuration="2.172272205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:36.168817377 +0000 UTC m=+2292.805636094" watchObservedRunningTime="2026-10-31 00:37:36.172272205 +0000 UTC m=+2292.809090921" Oct 31 00:37:36.765581 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:36.765542 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:37:36.765840 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:36.765808 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="def1055e-9b8b-4ad6-9203-b4f76835ec10" containerName="otc-container" containerID="cri-o://0237e7ff3200633864d2a182d48768e7e5079c871fb22f2fdec8c0d2c0111367" gracePeriod=30 Oct 31 00:37:39.114999 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.114977 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:39.163544 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.163459 5274 generic.go:356] "Generic (PLEG): container finished" podID="def1055e-9b8b-4ad6-9203-b4f76835ec10" containerID="0237e7ff3200633864d2a182d48768e7e5079c871fb22f2fdec8c0d2c0111367" exitCode=0 Oct 31 00:37:39.163544 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.163517 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:39.163743 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.163546 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"def1055e-9b8b-4ad6-9203-b4f76835ec10","Type":"ContainerDied","Data":"0237e7ff3200633864d2a182d48768e7e5079c871fb22f2fdec8c0d2c0111367"} Oct 31 00:37:39.163743 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.163598 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"def1055e-9b8b-4ad6-9203-b4f76835ec10","Type":"ContainerDied","Data":"97da7d0c3f0ff43243b420133020ddb9d5e8368fa824a3e44270b91c194d7a3c"} Oct 31 00:37:39.163743 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.163625 5274 scope.go:122] "RemoveContainer" containerID="0237e7ff3200633864d2a182d48768e7e5079c871fb22f2fdec8c0d2c0111367" Oct 31 00:37:39.175724 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.175708 5274 scope.go:122] "RemoveContainer" containerID="0237e7ff3200633864d2a182d48768e7e5079c871fb22f2fdec8c0d2c0111367" Oct 31 00:37:39.176014 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:37:39.175994 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0237e7ff3200633864d2a182d48768e7e5079c871fb22f2fdec8c0d2c0111367\": container with ID starting with 0237e7ff3200633864d2a182d48768e7e5079c871fb22f2fdec8c0d2c0111367 not found: ID does not exist" containerID="0237e7ff3200633864d2a182d48768e7e5079c871fb22f2fdec8c0d2c0111367" Oct 31 00:37:39.176097 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.176021 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0237e7ff3200633864d2a182d48768e7e5079c871fb22f2fdec8c0d2c0111367"} err="failed to get container status \"0237e7ff3200633864d2a182d48768e7e5079c871fb22f2fdec8c0d2c0111367\": rpc error: code = NotFound desc = could not find container \"0237e7ff3200633864d2a182d48768e7e5079c871fb22f2fdec8c0d2c0111367\": container with ID starting with 0237e7ff3200633864d2a182d48768e7e5079c871fb22f2fdec8c0d2c0111367 not found: ID does not exist" Oct 31 00:37:39.180234 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.180213 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/def1055e-9b8b-4ad6-9203-b4f76835ec10-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/def1055e-9b8b-4ad6-9203-b4f76835ec10-monitoring-tls-certs\") pod \"def1055e-9b8b-4ad6-9203-b4f76835ec10\" (UID: \"def1055e-9b8b-4ad6-9203-b4f76835ec10\") " Oct 31 00:37:39.180355 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.180272 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/def1055e-9b8b-4ad6-9203-b4f76835ec10-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/def1055e-9b8b-4ad6-9203-b4f76835ec10-otlp-tls-certs\") pod \"def1055e-9b8b-4ad6-9203-b4f76835ec10\" (UID: \"def1055e-9b8b-4ad6-9203-b4f76835ec10\") " Oct 31 00:37:39.180355 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.180335 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/def1055e-9b8b-4ad6-9203-b4f76835ec10-kube-api-access-d6z9l\" (UniqueName: \"kubernetes.io/projected/def1055e-9b8b-4ad6-9203-b4f76835ec10-kube-api-access-d6z9l\") pod \"def1055e-9b8b-4ad6-9203-b4f76835ec10\" (UID: \"def1055e-9b8b-4ad6-9203-b4f76835ec10\") " Oct 31 00:37:39.180470 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.180394 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/def1055e-9b8b-4ad6-9203-b4f76835ec10-otc-internal\" (UniqueName: \"kubernetes.io/configmap/def1055e-9b8b-4ad6-9203-b4f76835ec10-otc-internal\") pod \"def1055e-9b8b-4ad6-9203-b4f76835ec10\" (UID: \"def1055e-9b8b-4ad6-9203-b4f76835ec10\") " Oct 31 00:37:39.180470 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.180451 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/def1055e-9b8b-4ad6-9203-b4f76835ec10-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/def1055e-9b8b-4ad6-9203-b4f76835ec10-data-science-collector-ta-client-cert\") pod \"def1055e-9b8b-4ad6-9203-b4f76835ec10\" (UID: \"def1055e-9b8b-4ad6-9203-b4f76835ec10\") " Oct 31 00:37:39.180571 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.180482 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/def1055e-9b8b-4ad6-9203-b4f76835ec10-tls-certs\" (UniqueName: \"kubernetes.io/secret/def1055e-9b8b-4ad6-9203-b4f76835ec10-tls-certs\") pod \"def1055e-9b8b-4ad6-9203-b4f76835ec10\" (UID: \"def1055e-9b8b-4ad6-9203-b4f76835ec10\") " Oct 31 00:37:39.180862 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.180816 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/def1055e-9b8b-4ad6-9203-b4f76835ec10-otc-internal" pod "def1055e-9b8b-4ad6-9203-b4f76835ec10" (UID: "def1055e-9b8b-4ad6-9203-b4f76835ec10"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:39.182612 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.182576 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def1055e-9b8b-4ad6-9203-b4f76835ec10-kube-api-access-d6z9l" pod "def1055e-9b8b-4ad6-9203-b4f76835ec10" (UID: "def1055e-9b8b-4ad6-9203-b4f76835ec10"). InnerVolumeSpecName "kube-api-access-d6z9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:39.182703 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.182647 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/def1055e-9b8b-4ad6-9203-b4f76835ec10-tls-certs" pod "def1055e-9b8b-4ad6-9203-b4f76835ec10" (UID: "def1055e-9b8b-4ad6-9203-b4f76835ec10"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:39.182703 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.182657 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/def1055e-9b8b-4ad6-9203-b4f76835ec10-data-science-collector-ta-client-cert" pod "def1055e-9b8b-4ad6-9203-b4f76835ec10" (UID: "def1055e-9b8b-4ad6-9203-b4f76835ec10"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:39.182828 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.182813 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/def1055e-9b8b-4ad6-9203-b4f76835ec10-otlp-tls-certs" pod "def1055e-9b8b-4ad6-9203-b4f76835ec10" (UID: "def1055e-9b8b-4ad6-9203-b4f76835ec10"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:39.182878 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.182864 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/def1055e-9b8b-4ad6-9203-b4f76835ec10-monitoring-tls-certs" pod "def1055e-9b8b-4ad6-9203-b4f76835ec10" (UID: "def1055e-9b8b-4ad6-9203-b4f76835ec10"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:39.281656 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.281632 5274 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/def1055e-9b8b-4ad6-9203-b4f76835ec10-data-science-collector-ta-client-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:39.281656 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.281656 5274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def1055e-9b8b-4ad6-9203-b4f76835ec10-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:39.281801 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.281665 5274 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/def1055e-9b8b-4ad6-9203-b4f76835ec10-monitoring-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:39.281801 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.281674 5274 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/def1055e-9b8b-4ad6-9203-b4f76835ec10-otlp-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:39.281801 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.281682 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6z9l\" (UniqueName: \"kubernetes.io/projected/def1055e-9b8b-4ad6-9203-b4f76835ec10-kube-api-access-d6z9l\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:39.281801 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.281691 5274 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/def1055e-9b8b-4ad6-9203-b4f76835ec10-otc-internal\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:39.491070 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.491042 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:37:39.495169 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.495146 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:37:39.521376 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.521351 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:37:39.521716 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.521703 5274 cpu_manager.go:414] "containerMap: removing container" podUID="def1055e-9b8b-4ad6-9203-b4f76835ec10" containerName="otc-container" Oct 31 00:37:39.521762 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.521716 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="def1055e-9b8b-4ad6-9203-b4f76835ec10" containerName="otc-container" Oct 31 00:37:39.521810 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.521791 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="def1055e-9b8b-4ad6-9203-b4f76835ec10" containerName="otc-container" Oct 31 00:37:39.526223 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.526205 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:39.532604 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.532581 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:37:39.535067 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.534976 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-cfb2n\"" Oct 31 00:37:39.535172 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.535084 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:37:39.535245 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.535184 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:37:39.535470 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.535315 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:37:39.535900 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.535880 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-6d9d4583\"" Oct 31 00:37:39.538174 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.538154 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:37:39.583749 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.583726 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:39.583875 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.583756 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:39.583875 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.583792 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:39.583990 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.583883 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:39.583990 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.583949 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:39.584095 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.584053 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6smc\" (UniqueName: \"kubernetes.io/projected/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-kube-api-access-n6smc\") pod \"data-science-collector-collector-0\" (UID: \"59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:39.685004 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.684938 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:39.685175 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.685034 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:39.685175 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.685078 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6smc\" (UniqueName: \"kubernetes.io/projected/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-kube-api-access-n6smc\") pod \"data-science-collector-collector-0\" (UID: \"59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:39.685175 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.685134 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:39.685175 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.685159 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:39.685407 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.685215 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:39.686012 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.685980 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:39.687589 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.687563 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:39.687762 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.687737 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:39.687845 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.687803 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:39.687845 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.687825 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:39.696938 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.696919 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6smc\" (UniqueName: \"kubernetes.io/projected/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-kube-api-access-n6smc\") pod \"data-science-collector-collector-0\" (UID: \"59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:39.841196 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:39.841144 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:40.114607 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:40.114532 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:37:40.118681 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:37:40.118650 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b6a1ea_1b5a_466c_bcef_874e7e3a2cfa.slice/crio-5db0a286d069bfb095e823a64cdcce7bf6035bc993fa54b3f3d75df18a3b6435 WatchSource:0}: Error finding container 5db0a286d069bfb095e823a64cdcce7bf6035bc993fa54b3f3d75df18a3b6435: Status 404 returned error can't find the container with id 5db0a286d069bfb095e823a64cdcce7bf6035bc993fa54b3f3d75df18a3b6435 Oct 31 00:37:40.121538 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:40.121497 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:37:40.121631 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:40.121577 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:37:40.170826 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:40.170798 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa","Type":"ContainerStarted","Data":"5db0a286d069bfb095e823a64cdcce7bf6035bc993fa54b3f3d75df18a3b6435"} Oct 31 00:37:41.177137 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:41.177091 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa","Type":"ContainerStarted","Data":"29d595f39fe62e580fbb2b439a606e03f6ce21b84ed0a9743ff0a836d0e2df45"} Oct 31 00:37:41.198473 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:41.198427 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.198412923 podStartE2EDuration="2.198412923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:41.196661186 +0000 UTC m=+2297.833479902" watchObservedRunningTime="2026-10-31 00:37:41.198412923 +0000 UTC m=+2297.835231638" Oct 31 00:37:41.482011 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:41.481908 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="def1055e-9b8b-4ad6-9203-b4f76835ec10" path="/var/lib/kubelet/pods/def1055e-9b8b-4ad6-9203-b4f76835ec10/volumes" Oct 31 00:37:47.028333 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:47.028302 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-gcj9x"] Oct 31 00:37:47.028778 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:47.028514 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-gcj9x" podUID="7f0518ab-c116-4b78-9079-9f45d67d1f1a" containerName="korrel8r" containerID="cri-o://03a41eed5f69cfee720cac0f538cc5d029646fed4ec8fd63a4f41975c6614d48" gracePeriod=30 Oct 31 00:37:47.030656 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:47.030633 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-gcj9x" Oct 31 00:37:47.210253 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:47.210223 5274 generic.go:356] "Generic (PLEG): container finished" podID="7f0518ab-c116-4b78-9079-9f45d67d1f1a" containerID="03a41eed5f69cfee720cac0f538cc5d029646fed4ec8fd63a4f41975c6614d48" exitCode=0 Oct 31 00:37:47.210407 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:47.210273 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-gcj9x" event={"ID":"7f0518ab-c116-4b78-9079-9f45d67d1f1a","Type":"ContainerDied","Data":"03a41eed5f69cfee720cac0f538cc5d029646fed4ec8fd63a4f41975c6614d48"} Oct 31 00:37:47.277408 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:47.277383 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-gcj9x" Oct 31 00:37:47.350880 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:47.350812 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f0518ab-c116-4b78-9079-9f45d67d1f1a-tls\" (UniqueName: \"kubernetes.io/secret/7f0518ab-c116-4b78-9079-9f45d67d1f1a-tls\") pod \"7f0518ab-c116-4b78-9079-9f45d67d1f1a\" (UID: \"7f0518ab-c116-4b78-9079-9f45d67d1f1a\") " Oct 31 00:37:47.351055 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:47.350916 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f0518ab-c116-4b78-9079-9f45d67d1f1a-kube-api-access-zp8xj\" (UniqueName: \"kubernetes.io/projected/7f0518ab-c116-4b78-9079-9f45d67d1f1a-kube-api-access-zp8xj\") pod \"7f0518ab-c116-4b78-9079-9f45d67d1f1a\" (UID: \"7f0518ab-c116-4b78-9079-9f45d67d1f1a\") " Oct 31 00:37:47.351055 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:47.350942 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f0518ab-c116-4b78-9079-9f45d67d1f1a-config\" (UniqueName: \"kubernetes.io/configmap/7f0518ab-c116-4b78-9079-9f45d67d1f1a-config\") pod \"7f0518ab-c116-4b78-9079-9f45d67d1f1a\" (UID: \"7f0518ab-c116-4b78-9079-9f45d67d1f1a\") " Oct 31 00:37:47.351290 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:47.351271 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f0518ab-c116-4b78-9079-9f45d67d1f1a-config" pod "7f0518ab-c116-4b78-9079-9f45d67d1f1a" (UID: "7f0518ab-c116-4b78-9079-9f45d67d1f1a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:47.353047 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:47.352933 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f0518ab-c116-4b78-9079-9f45d67d1f1a-kube-api-access-zp8xj" pod "7f0518ab-c116-4b78-9079-9f45d67d1f1a" (UID: "7f0518ab-c116-4b78-9079-9f45d67d1f1a"). InnerVolumeSpecName "kube-api-access-zp8xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:47.353047 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:47.352968 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f0518ab-c116-4b78-9079-9f45d67d1f1a-tls" pod "7f0518ab-c116-4b78-9079-9f45d67d1f1a" (UID: "7f0518ab-c116-4b78-9079-9f45d67d1f1a"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:47.452284 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:47.452256 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp8xj\" (UniqueName: \"kubernetes.io/projected/7f0518ab-c116-4b78-9079-9f45d67d1f1a-kube-api-access-zp8xj\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:47.452284 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:47.452284 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7f0518ab-c116-4b78-9079-9f45d67d1f1a-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:47.452471 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:47.452297 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/7f0518ab-c116-4b78-9079-9f45d67d1f1a-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:48.216971 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:48.216934 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-gcj9x" Oct 31 00:37:48.217450 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:48.216981 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-gcj9x" event={"ID":"7f0518ab-c116-4b78-9079-9f45d67d1f1a","Type":"ContainerDied","Data":"415090a30b6afefa24857f0ea423958ccd288b7ae760e29a4549a86c4555b78e"} Oct 31 00:37:48.217450 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:48.217037 5274 scope.go:122] "RemoveContainer" containerID="03a41eed5f69cfee720cac0f538cc5d029646fed4ec8fd63a4f41975c6614d48" Oct 31 00:37:48.235288 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:48.235263 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-gcj9x"] Oct 31 00:37:48.240167 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:48.240146 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-gcj9x"] Oct 31 00:37:49.481523 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:49.481494 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f0518ab-c116-4b78-9079-9f45d67d1f1a" path="/var/lib/kubelet/pods/7f0518ab-c116-4b78-9079-9f45d67d1f1a/volumes" Oct 31 00:37:50.899883 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:37:50.899849 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:37:51.312079 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:51.311997 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:37:51.312249 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:51.312208 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa" containerName="otc-container" containerID="cri-o://29d595f39fe62e580fbb2b439a606e03f6ce21b84ed0a9743ff0a836d0e2df45" gracePeriod=30 Oct 31 00:37:51.561589 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:51.561562 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:51.687950 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:51.687916 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-otc-internal\" (UniqueName: \"kubernetes.io/configmap/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-otc-internal\") pod \"59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa\" (UID: \"59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa\") " Oct 31 00:37:51.688162 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:51.688061 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-tls-certs\" (UniqueName: \"kubernetes.io/secret/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-tls-certs\") pod \"59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa\" (UID: \"59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa\") " Oct 31 00:37:51.688162 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:51.688107 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-otlp-tls-certs\") pod \"59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa\" (UID: \"59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa\") " Oct 31 00:37:51.688162 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:51.688143 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-kube-api-access-n6smc\" (UniqueName: \"kubernetes.io/projected/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-kube-api-access-n6smc\") pod \"59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa\" (UID: \"59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa\") " Oct 31 00:37:51.688327 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:51.688198 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-data-science-collector-ta-client-cert\") pod \"59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa\" (UID: \"59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa\") " Oct 31 00:37:51.688327 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:51.688242 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-monitoring-tls-certs\") pod \"59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa\" (UID: \"59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa\") " Oct 31 00:37:51.688436 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:51.688323 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-otc-internal" pod "59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa" (UID: "59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:51.688609 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:51.688580 5274 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-otc-internal\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:51.690427 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:51.690402 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-monitoring-tls-certs" pod "59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa" (UID: "59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:51.690553 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:51.690426 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-kube-api-access-n6smc" pod "59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa" (UID: "59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa"). InnerVolumeSpecName "kube-api-access-n6smc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:51.690553 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:51.690426 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-data-science-collector-ta-client-cert" pod "59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa" (UID: "59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:51.690669 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:51.690655 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-tls-certs" pod "59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa" (UID: "59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:51.690721 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:51.690673 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-otlp-tls-certs" pod "59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa" (UID: "59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:51.789007 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:51.788976 5274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:51.789007 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:51.788999 5274 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-otlp-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:51.789007 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:51.789009 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6smc\" (UniqueName: \"kubernetes.io/projected/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-kube-api-access-n6smc\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:51.789224 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:51.789021 5274 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-data-science-collector-ta-client-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:51.789224 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:51.789030 5274 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa-monitoring-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:52.240040 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.240004 5274 generic.go:356] "Generic (PLEG): container finished" podID="59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa" containerID="29d595f39fe62e580fbb2b439a606e03f6ce21b84ed0a9743ff0a836d0e2df45" exitCode=0 Oct 31 00:37:52.240437 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.240067 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:52.240437 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.240088 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa","Type":"ContainerDied","Data":"29d595f39fe62e580fbb2b439a606e03f6ce21b84ed0a9743ff0a836d0e2df45"} Oct 31 00:37:52.240437 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.240153 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa","Type":"ContainerDied","Data":"5db0a286d069bfb095e823a64cdcce7bf6035bc993fa54b3f3d75df18a3b6435"} Oct 31 00:37:52.240437 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.240179 5274 scope.go:122] "RemoveContainer" containerID="29d595f39fe62e580fbb2b439a606e03f6ce21b84ed0a9743ff0a836d0e2df45" Oct 31 00:37:52.249462 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.249445 5274 scope.go:122] "RemoveContainer" containerID="29d595f39fe62e580fbb2b439a606e03f6ce21b84ed0a9743ff0a836d0e2df45" Oct 31 00:37:52.249725 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:37:52.249708 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29d595f39fe62e580fbb2b439a606e03f6ce21b84ed0a9743ff0a836d0e2df45\": container with ID starting with 29d595f39fe62e580fbb2b439a606e03f6ce21b84ed0a9743ff0a836d0e2df45 not found: ID does not exist" containerID="29d595f39fe62e580fbb2b439a606e03f6ce21b84ed0a9743ff0a836d0e2df45" Oct 31 00:37:52.249786 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.249732 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29d595f39fe62e580fbb2b439a606e03f6ce21b84ed0a9743ff0a836d0e2df45"} err="failed to get container status \"29d595f39fe62e580fbb2b439a606e03f6ce21b84ed0a9743ff0a836d0e2df45\": rpc error: code = NotFound desc = could not find container \"29d595f39fe62e580fbb2b439a606e03f6ce21b84ed0a9743ff0a836d0e2df45\": container with ID starting with 29d595f39fe62e580fbb2b439a606e03f6ce21b84ed0a9743ff0a836d0e2df45 not found: ID does not exist" Oct 31 00:37:52.262442 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.262421 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:37:52.265852 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.265832 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:37:52.292044 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.292020 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:37:52.292383 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.292371 5274 cpu_manager.go:414] "containerMap: removing container" podUID="59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa" containerName="otc-container" Oct 31 00:37:52.292428 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.292384 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa" containerName="otc-container" Oct 31 00:37:52.292428 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.292403 5274 cpu_manager.go:414] "containerMap: removing container" podUID="7f0518ab-c116-4b78-9079-9f45d67d1f1a" containerName="korrel8r" Oct 31 00:37:52.292428 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.292409 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f0518ab-c116-4b78-9079-9f45d67d1f1a" containerName="korrel8r" Oct 31 00:37:52.292518 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.292471 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa" containerName="otc-container" Oct 31 00:37:52.292518 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.292481 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f0518ab-c116-4b78-9079-9f45d67d1f1a" containerName="korrel8r" Oct 31 00:37:52.297011 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.296992 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:52.301350 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.301328 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:37:52.301653 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.301627 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:37:52.301760 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.301654 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:37:52.301760 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.301679 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:37:52.301760 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.301659 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-cfb2n\"" Oct 31 00:37:52.301760 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.301693 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:37:52.307617 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.307596 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:37:52.394259 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.394230 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pql28\" (UniqueName: \"kubernetes.io/projected/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-kube-api-access-pql28\") pod \"data-science-collector-collector-0\" (UID: \"7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:52.394259 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.394260 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:52.394440 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.394284 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:52.394440 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.394414 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:52.394509 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.394466 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:52.394509 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.394493 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:52.495718 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.495644 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pql28\" (UniqueName: \"kubernetes.io/projected/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-kube-api-access-pql28\") pod \"data-science-collector-collector-0\" (UID: \"7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:52.495718 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.495676 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:52.495718 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.495697 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:52.496004 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.495727 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:52.496004 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.495755 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:52.496004 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.495779 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:52.496565 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.496509 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:52.498338 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.498315 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:52.498447 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.498346 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:52.498513 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.498463 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:52.498565 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.498532 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:52.507842 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.507823 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pql28\" (UniqueName: \"kubernetes.io/projected/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-kube-api-access-pql28\") pod \"data-science-collector-collector-0\" (UID: \"7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:52.612033 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.612011 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:52.886298 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.886270 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:37:52.887737 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:37:52.887711 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cf5b299_5a4a_45b2_b8dc_a1124ee37cb8.slice/crio-a2e75b53c747fb5930801931a249d6ec607e2b24d86f17d2fd386f509b772e89 WatchSource:0}: Error finding container a2e75b53c747fb5930801931a249d6ec607e2b24d86f17d2fd386f509b772e89: Status 404 returned error can't find the container with id a2e75b53c747fb5930801931a249d6ec607e2b24d86f17d2fd386f509b772e89 Oct 31 00:37:52.890006 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.889939 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:37:52.890093 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:52.890032 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:37:53.220045 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.219971 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k28gg"] Oct 31 00:37:53.220197 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.220175 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k28gg" podUID="3dc6465e-bbd3-4e25-b4c6-d28231ef86d6" containerName="kube-rbac-proxy" containerID="cri-o://b06fd2a6db91e81b02491057efef9f2ef2c1e2259d982ae58f91708b94c12f57" gracePeriod=30 Oct 31 00:37:53.237760 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.237734 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-gcwfl"] Oct 31 00:37:53.237916 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.237898 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gcwfl" podUID="53262348-aaeb-4648-8b93-3b4c40f3ebc2" containerName="korrel8r" containerID="cri-o://7927106a65370a34199843d6da7a61c19272426c8d67f88f57e0db714d50a0ac" gracePeriod=30 Oct 31 00:37:53.251310 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.251285 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8","Type":"ContainerStarted","Data":"7a0b85fedb4f4506a47c880b497e9dd62394da4054f5f2108379f20378dd6431"} Oct 31 00:37:53.251634 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.251333 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8","Type":"ContainerStarted","Data":"a2e75b53c747fb5930801931a249d6ec607e2b24d86f17d2fd386f509b772e89"} Oct 31 00:37:53.283533 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.283491 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.283479244 podStartE2EDuration="1.283479244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:53.282279989 +0000 UTC m=+2309.919098707" watchObservedRunningTime="2026-10-31 00:37:53.283479244 +0000 UTC m=+2309.920297960" Oct 31 00:37:53.426754 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.426711 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:37:53.427021 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.426994 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="1294ed53-4e5a-4105-8acd-b2a04cce8df9" containerName="alertmanager" containerID="cri-o://8336c41d20990947f1b50c861028d66de9cc789fdb475a52a5d50d2e6033a51b" gracePeriod=120 Oct 31 00:37:53.427562 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.427378 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="1294ed53-4e5a-4105-8acd-b2a04cce8df9" containerName="config-reloader" containerID="cri-o://74f50f1c5d3b2e1f48497e1aee716fbc62b70202794e48526ecf56578c8a6837" gracePeriod=120 Oct 31 00:37:53.444188 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.444152 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:37:53.444481 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.444456 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="638c279f-efbc-45ec-ac21-5d47e6def796" containerName="prometheus" containerID="cri-o://71fa5393b769366c416449f82bd3143db69a43e18de39b3d7f9c9d831540a48a" gracePeriod=600 Oct 31 00:37:53.445114 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.444828 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="638c279f-efbc-45ec-ac21-5d47e6def796" containerName="config-reloader" containerID="cri-o://e8e42586c9f22feb9b15b05543e5538500b873b0c2c889add6cec90c662cd0da" gracePeriod=600 Oct 31 00:37:53.445114 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.445016 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="638c279f-efbc-45ec-ac21-5d47e6def796" containerName="thanos-sidecar" containerID="cri-o://2b24cbad42e3d9e26b15f0c49f8bbd9c7f6be7c5ff9b4770d486552f0e95db4a" gracePeriod=600 Oct 31 00:37:53.485674 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.485646 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa" path="/var/lib/kubelet/pods/59b6a1ea-1b5a-466c-bcef-874e7e3a2cfa/volumes" Oct 31 00:37:53.588165 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.588135 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:37:53.605162 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.605129 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k28gg" Oct 31 00:37:53.607347 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.607324 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gcwfl" Oct 31 00:37:53.708434 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.708399 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53262348-aaeb-4648-8b93-3b4c40f3ebc2-config\" (UniqueName: \"kubernetes.io/configmap/53262348-aaeb-4648-8b93-3b4c40f3ebc2-config\") pod \"53262348-aaeb-4648-8b93-3b4c40f3ebc2\" (UID: \"53262348-aaeb-4648-8b93-3b4c40f3ebc2\") " Oct 31 00:37:53.708592 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.708450 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-prometheus-client-cert\") pod \"3dc6465e-bbd3-4e25-b4c6-d28231ef86d6\" (UID: \"3dc6465e-bbd3-4e25-b4c6-d28231ef86d6\") " Oct 31 00:37:53.708592 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.708540 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-kube-api-access-7nfgh\" (UniqueName: \"kubernetes.io/projected/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-kube-api-access-7nfgh\") pod \"3dc6465e-bbd3-4e25-b4c6-d28231ef86d6\" (UID: \"3dc6465e-bbd3-4e25-b4c6-d28231ef86d6\") " Oct 31 00:37:53.708592 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.708582 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-kube-rbac-proxy-config\") pod \"3dc6465e-bbd3-4e25-b4c6-d28231ef86d6\" (UID: \"3dc6465e-bbd3-4e25-b4c6-d28231ef86d6\") " Oct 31 00:37:53.708770 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.708617 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-tls\" (UniqueName: \"kubernetes.io/secret/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-tls\") pod \"3dc6465e-bbd3-4e25-b4c6-d28231ef86d6\" (UID: \"3dc6465e-bbd3-4e25-b4c6-d28231ef86d6\") " Oct 31 00:37:53.708770 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.708652 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53262348-aaeb-4648-8b93-3b4c40f3ebc2-tls\" (UniqueName: \"kubernetes.io/secret/53262348-aaeb-4648-8b93-3b4c40f3ebc2-tls\") pod \"53262348-aaeb-4648-8b93-3b4c40f3ebc2\" (UID: \"53262348-aaeb-4648-8b93-3b4c40f3ebc2\") " Oct 31 00:37:53.708770 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.708685 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-prometheus-ca\") pod \"3dc6465e-bbd3-4e25-b4c6-d28231ef86d6\" (UID: \"3dc6465e-bbd3-4e25-b4c6-d28231ef86d6\") " Oct 31 00:37:53.708770 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.708736 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53262348-aaeb-4648-8b93-3b4c40f3ebc2-kube-api-access-gzc4t\" (UniqueName: \"kubernetes.io/projected/53262348-aaeb-4648-8b93-3b4c40f3ebc2-kube-api-access-gzc4t\") pod \"53262348-aaeb-4648-8b93-3b4c40f3ebc2\" (UID: \"53262348-aaeb-4648-8b93-3b4c40f3ebc2\") " Oct 31 00:37:53.708978 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.708774 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53262348-aaeb-4648-8b93-3b4c40f3ebc2-config" pod "53262348-aaeb-4648-8b93-3b4c40f3ebc2" (UID: "53262348-aaeb-4648-8b93-3b4c40f3ebc2"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:53.709044 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.709028 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/53262348-aaeb-4648-8b93-3b4c40f3ebc2-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:53.709896 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.709868 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-prometheus-ca" pod "3dc6465e-bbd3-4e25-b4c6-d28231ef86d6" (UID: "3dc6465e-bbd3-4e25-b4c6-d28231ef86d6"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:53.712737 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.712711 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53262348-aaeb-4648-8b93-3b4c40f3ebc2-kube-api-access-gzc4t" pod "53262348-aaeb-4648-8b93-3b4c40f3ebc2" (UID: "53262348-aaeb-4648-8b93-3b4c40f3ebc2"). InnerVolumeSpecName "kube-api-access-gzc4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:53.712832 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.712774 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-prometheus-client-cert" pod "3dc6465e-bbd3-4e25-b4c6-d28231ef86d6" (UID: "3dc6465e-bbd3-4e25-b4c6-d28231ef86d6"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:53.712832 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.712786 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-tls" pod "3dc6465e-bbd3-4e25-b4c6-d28231ef86d6" (UID: "3dc6465e-bbd3-4e25-b4c6-d28231ef86d6"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:53.712832 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.712786 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-kube-api-access-7nfgh" pod "3dc6465e-bbd3-4e25-b4c6-d28231ef86d6" (UID: "3dc6465e-bbd3-4e25-b4c6-d28231ef86d6"). InnerVolumeSpecName "kube-api-access-7nfgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:53.713024 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.712851 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-kube-rbac-proxy-config" pod "3dc6465e-bbd3-4e25-b4c6-d28231ef86d6" (UID: "3dc6465e-bbd3-4e25-b4c6-d28231ef86d6"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:53.713384 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.713364 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53262348-aaeb-4648-8b93-3b4c40f3ebc2-tls" pod "53262348-aaeb-4648-8b93-3b4c40f3ebc2" (UID: "53262348-aaeb-4648-8b93-3b4c40f3ebc2"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:53.759977 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.759891 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="638c279f-efbc-45ec-ac21-5d47e6def796" containerName="prometheus" probeResult="failure" output="Get \"https://10.134.0.90:9090/-/ready\": dial tcp 10.134.0.90:9090: connect: connection refused" Oct 31 00:37:53.809873 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.809842 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzc4t\" (UniqueName: \"kubernetes.io/projected/53262348-aaeb-4648-8b93-3b4c40f3ebc2-kube-api-access-gzc4t\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:53.809873 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.809871 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-prometheus-client-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:53.810089 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.809884 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nfgh\" (UniqueName: \"kubernetes.io/projected/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-kube-api-access-7nfgh\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:53.810089 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.809901 5274 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-kube-rbac-proxy-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:53.810089 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.809913 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:53.810089 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.809925 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/53262348-aaeb-4648-8b93-3b4c40f3ebc2-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:53.810089 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:53.809936 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6-prometheus-ca\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:54.181767 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.181743 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:37:54.258882 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.258845 5274 generic.go:356] "Generic (PLEG): container finished" podID="1294ed53-4e5a-4105-8acd-b2a04cce8df9" containerID="74f50f1c5d3b2e1f48497e1aee716fbc62b70202794e48526ecf56578c8a6837" exitCode=0 Oct 31 00:37:54.259325 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.258918 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"1294ed53-4e5a-4105-8acd-b2a04cce8df9","Type":"ContainerDied","Data":"74f50f1c5d3b2e1f48497e1aee716fbc62b70202794e48526ecf56578c8a6837"} Oct 31 00:37:54.260249 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.260228 5274 generic.go:356] "Generic (PLEG): container finished" podID="53262348-aaeb-4648-8b93-3b4c40f3ebc2" containerID="7927106a65370a34199843d6da7a61c19272426c8d67f88f57e0db714d50a0ac" exitCode=0 Oct 31 00:37:54.260375 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.260281 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gcwfl" Oct 31 00:37:54.260375 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.260316 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gcwfl" event={"ID":"53262348-aaeb-4648-8b93-3b4c40f3ebc2","Type":"ContainerDied","Data":"7927106a65370a34199843d6da7a61c19272426c8d67f88f57e0db714d50a0ac"} Oct 31 00:37:54.260375 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.260365 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-gcwfl" event={"ID":"53262348-aaeb-4648-8b93-3b4c40f3ebc2","Type":"ContainerDied","Data":"3aa9ba7edbeb8c5cf819c502637ecacbffbc7d15b7e38a394f58baaad2ecc5a6"} Oct 31 00:37:54.260531 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.260393 5274 scope.go:122] "RemoveContainer" containerID="7927106a65370a34199843d6da7a61c19272426c8d67f88f57e0db714d50a0ac" Oct 31 00:37:54.261521 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.261501 5274 generic.go:356] "Generic (PLEG): container finished" podID="3dc6465e-bbd3-4e25-b4c6-d28231ef86d6" containerID="b06fd2a6db91e81b02491057efef9f2ef2c1e2259d982ae58f91708b94c12f57" exitCode=0 Oct 31 00:37:54.261636 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.261534 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k28gg" event={"ID":"3dc6465e-bbd3-4e25-b4c6-d28231ef86d6","Type":"ContainerDied","Data":"b06fd2a6db91e81b02491057efef9f2ef2c1e2259d982ae58f91708b94c12f57"} Oct 31 00:37:54.261636 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.261577 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k28gg" Oct 31 00:37:54.261636 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.261589 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k28gg" event={"ID":"3dc6465e-bbd3-4e25-b4c6-d28231ef86d6","Type":"ContainerDied","Data":"3097a10d75a789ea4ee45c47b2850c6cc4bc0f7aa34f225ee973efcc27c2c1cc"} Oct 31 00:37:54.264072 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.264048 5274 generic.go:356] "Generic (PLEG): container finished" podID="638c279f-efbc-45ec-ac21-5d47e6def796" containerID="2b24cbad42e3d9e26b15f0c49f8bbd9c7f6be7c5ff9b4770d486552f0e95db4a" exitCode=0 Oct 31 00:37:54.264072 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.264071 5274 generic.go:356] "Generic (PLEG): container finished" podID="638c279f-efbc-45ec-ac21-5d47e6def796" containerID="e8e42586c9f22feb9b15b05543e5538500b873b0c2c889add6cec90c662cd0da" exitCode=0 Oct 31 00:37:54.264243 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.264084 5274 generic.go:356] "Generic (PLEG): container finished" podID="638c279f-efbc-45ec-ac21-5d47e6def796" containerID="71fa5393b769366c416449f82bd3143db69a43e18de39b3d7f9c9d831540a48a" exitCode=0 Oct 31 00:37:54.264243 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.264132 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"638c279f-efbc-45ec-ac21-5d47e6def796","Type":"ContainerDied","Data":"2b24cbad42e3d9e26b15f0c49f8bbd9c7f6be7c5ff9b4770d486552f0e95db4a"} Oct 31 00:37:54.264243 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.264191 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"638c279f-efbc-45ec-ac21-5d47e6def796","Type":"ContainerDied","Data":"e8e42586c9f22feb9b15b05543e5538500b873b0c2c889add6cec90c662cd0da"} Oct 31 00:37:54.264243 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.264217 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"638c279f-efbc-45ec-ac21-5d47e6def796","Type":"ContainerDied","Data":"71fa5393b769366c416449f82bd3143db69a43e18de39b3d7f9c9d831540a48a"} Oct 31 00:37:54.264243 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.264241 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"638c279f-efbc-45ec-ac21-5d47e6def796","Type":"ContainerDied","Data":"6a39b169d778ef0e23ea7c0b5afa1f3b24e133c4714cf7432018ce5f90f84d6c"} Oct 31 00:37:54.264480 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.264242 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:37:54.264480 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.264345 5274 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="redhat-ods-monitoring/data-science-collector-collector-0" secret="" err="secret \"data-science-collector-collector-dockercfg-cfb2n\" not found" Oct 31 00:37:54.270862 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.270844 5274 scope.go:122] "RemoveContainer" containerID="7927106a65370a34199843d6da7a61c19272426c8d67f88f57e0db714d50a0ac" Oct 31 00:37:54.271180 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:37:54.271159 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7927106a65370a34199843d6da7a61c19272426c8d67f88f57e0db714d50a0ac\": container with ID starting with 7927106a65370a34199843d6da7a61c19272426c8d67f88f57e0db714d50a0ac not found: ID does not exist" containerID="7927106a65370a34199843d6da7a61c19272426c8d67f88f57e0db714d50a0ac" Oct 31 00:37:54.271268 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.271189 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7927106a65370a34199843d6da7a61c19272426c8d67f88f57e0db714d50a0ac"} err="failed to get container status \"7927106a65370a34199843d6da7a61c19272426c8d67f88f57e0db714d50a0ac\": rpc error: code = NotFound desc = could not find container \"7927106a65370a34199843d6da7a61c19272426c8d67f88f57e0db714d50a0ac\": container with ID starting with 7927106a65370a34199843d6da7a61c19272426c8d67f88f57e0db714d50a0ac not found: ID does not exist" Oct 31 00:37:54.271268 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.271209 5274 scope.go:122] "RemoveContainer" containerID="b06fd2a6db91e81b02491057efef9f2ef2c1e2259d982ae58f91708b94c12f57" Oct 31 00:37:54.279612 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.279598 5274 scope.go:122] "RemoveContainer" containerID="b06fd2a6db91e81b02491057efef9f2ef2c1e2259d982ae58f91708b94c12f57" Oct 31 00:37:54.279839 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:37:54.279822 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b06fd2a6db91e81b02491057efef9f2ef2c1e2259d982ae58f91708b94c12f57\": container with ID starting with b06fd2a6db91e81b02491057efef9f2ef2c1e2259d982ae58f91708b94c12f57 not found: ID does not exist" containerID="b06fd2a6db91e81b02491057efef9f2ef2c1e2259d982ae58f91708b94c12f57" Oct 31 00:37:54.279879 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.279845 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b06fd2a6db91e81b02491057efef9f2ef2c1e2259d982ae58f91708b94c12f57"} err="failed to get container status \"b06fd2a6db91e81b02491057efef9f2ef2c1e2259d982ae58f91708b94c12f57\": rpc error: code = NotFound desc = could not find container \"b06fd2a6db91e81b02491057efef9f2ef2c1e2259d982ae58f91708b94c12f57\": container with ID starting with b06fd2a6db91e81b02491057efef9f2ef2c1e2259d982ae58f91708b94c12f57 not found: ID does not exist" Oct 31 00:37:54.279879 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.279860 5274 scope.go:122] "RemoveContainer" containerID="2b24cbad42e3d9e26b15f0c49f8bbd9c7f6be7c5ff9b4770d486552f0e95db4a" Oct 31 00:37:54.287847 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.287386 5274 scope.go:122] "RemoveContainer" containerID="e8e42586c9f22feb9b15b05543e5538500b873b0c2c889add6cec90c662cd0da" Oct 31 00:37:54.289189 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.289168 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k28gg"] Oct 31 00:37:54.293430 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.293412 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-k28gg"] Oct 31 00:37:54.296179 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.296165 5274 scope.go:122] "RemoveContainer" containerID="71fa5393b769366c416449f82bd3143db69a43e18de39b3d7f9c9d831540a48a" Oct 31 00:37:54.304159 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.303569 5274 scope.go:122] "RemoveContainer" containerID="47de2c4275cc951b65c9360d45be4b57ab1bf2ce5a8c29d55731c98b95b5c5e7" Oct 31 00:37:54.305552 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.305532 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-gcwfl"] Oct 31 00:37:54.309994 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.309973 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-gcwfl"] Oct 31 00:37:54.311762 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.311746 5274 scope.go:122] "RemoveContainer" containerID="2b24cbad42e3d9e26b15f0c49f8bbd9c7f6be7c5ff9b4770d486552f0e95db4a" Oct 31 00:37:54.312045 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:37:54.312027 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b24cbad42e3d9e26b15f0c49f8bbd9c7f6be7c5ff9b4770d486552f0e95db4a\": container with ID starting with 2b24cbad42e3d9e26b15f0c49f8bbd9c7f6be7c5ff9b4770d486552f0e95db4a not found: ID does not exist" containerID="2b24cbad42e3d9e26b15f0c49f8bbd9c7f6be7c5ff9b4770d486552f0e95db4a" Oct 31 00:37:54.312122 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.312057 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b24cbad42e3d9e26b15f0c49f8bbd9c7f6be7c5ff9b4770d486552f0e95db4a"} err="failed to get container status \"2b24cbad42e3d9e26b15f0c49f8bbd9c7f6be7c5ff9b4770d486552f0e95db4a\": rpc error: code = NotFound desc = could not find container \"2b24cbad42e3d9e26b15f0c49f8bbd9c7f6be7c5ff9b4770d486552f0e95db4a\": container with ID starting with 2b24cbad42e3d9e26b15f0c49f8bbd9c7f6be7c5ff9b4770d486552f0e95db4a not found: ID does not exist" Oct 31 00:37:54.312122 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.312081 5274 scope.go:122] "RemoveContainer" containerID="e8e42586c9f22feb9b15b05543e5538500b873b0c2c889add6cec90c662cd0da" Oct 31 00:37:54.312344 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:37:54.312327 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8e42586c9f22feb9b15b05543e5538500b873b0c2c889add6cec90c662cd0da\": container with ID starting with e8e42586c9f22feb9b15b05543e5538500b873b0c2c889add6cec90c662cd0da not found: ID does not exist" containerID="e8e42586c9f22feb9b15b05543e5538500b873b0c2c889add6cec90c662cd0da" Oct 31 00:37:54.312383 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.312350 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8e42586c9f22feb9b15b05543e5538500b873b0c2c889add6cec90c662cd0da"} err="failed to get container status \"e8e42586c9f22feb9b15b05543e5538500b873b0c2c889add6cec90c662cd0da\": rpc error: code = NotFound desc = could not find container \"e8e42586c9f22feb9b15b05543e5538500b873b0c2c889add6cec90c662cd0da\": container with ID starting with e8e42586c9f22feb9b15b05543e5538500b873b0c2c889add6cec90c662cd0da not found: ID does not exist" Oct 31 00:37:54.312383 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.312364 5274 scope.go:122] "RemoveContainer" containerID="71fa5393b769366c416449f82bd3143db69a43e18de39b3d7f9c9d831540a48a" Oct 31 00:37:54.312585 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:37:54.312568 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71fa5393b769366c416449f82bd3143db69a43e18de39b3d7f9c9d831540a48a\": container with ID starting with 71fa5393b769366c416449f82bd3143db69a43e18de39b3d7f9c9d831540a48a not found: ID does not exist" containerID="71fa5393b769366c416449f82bd3143db69a43e18de39b3d7f9c9d831540a48a" Oct 31 00:37:54.312656 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.312593 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71fa5393b769366c416449f82bd3143db69a43e18de39b3d7f9c9d831540a48a"} err="failed to get container status \"71fa5393b769366c416449f82bd3143db69a43e18de39b3d7f9c9d831540a48a\": rpc error: code = NotFound desc = could not find container \"71fa5393b769366c416449f82bd3143db69a43e18de39b3d7f9c9d831540a48a\": container with ID starting with 71fa5393b769366c416449f82bd3143db69a43e18de39b3d7f9c9d831540a48a not found: ID does not exist" Oct 31 00:37:54.312656 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.312613 5274 scope.go:122] "RemoveContainer" containerID="47de2c4275cc951b65c9360d45be4b57ab1bf2ce5a8c29d55731c98b95b5c5e7" Oct 31 00:37:54.312844 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:37:54.312828 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47de2c4275cc951b65c9360d45be4b57ab1bf2ce5a8c29d55731c98b95b5c5e7\": container with ID starting with 47de2c4275cc951b65c9360d45be4b57ab1bf2ce5a8c29d55731c98b95b5c5e7 not found: ID does not exist" containerID="47de2c4275cc951b65c9360d45be4b57ab1bf2ce5a8c29d55731c98b95b5c5e7" Oct 31 00:37:54.312882 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.312850 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47de2c4275cc951b65c9360d45be4b57ab1bf2ce5a8c29d55731c98b95b5c5e7"} err="failed to get container status \"47de2c4275cc951b65c9360d45be4b57ab1bf2ce5a8c29d55731c98b95b5c5e7\": rpc error: code = NotFound desc = could not find container \"47de2c4275cc951b65c9360d45be4b57ab1bf2ce5a8c29d55731c98b95b5c5e7\": container with ID starting with 47de2c4275cc951b65c9360d45be4b57ab1bf2ce5a8c29d55731c98b95b5c5e7 not found: ID does not exist" Oct 31 00:37:54.312882 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.312864 5274 scope.go:122] "RemoveContainer" containerID="2b24cbad42e3d9e26b15f0c49f8bbd9c7f6be7c5ff9b4770d486552f0e95db4a" Oct 31 00:37:54.313135 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.313116 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b24cbad42e3d9e26b15f0c49f8bbd9c7f6be7c5ff9b4770d486552f0e95db4a"} err="failed to get container status \"2b24cbad42e3d9e26b15f0c49f8bbd9c7f6be7c5ff9b4770d486552f0e95db4a\": rpc error: code = NotFound desc = could not find container \"2b24cbad42e3d9e26b15f0c49f8bbd9c7f6be7c5ff9b4770d486552f0e95db4a\": container with ID starting with 2b24cbad42e3d9e26b15f0c49f8bbd9c7f6be7c5ff9b4770d486552f0e95db4a not found: ID does not exist" Oct 31 00:37:54.313208 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.313136 5274 scope.go:122] "RemoveContainer" containerID="e8e42586c9f22feb9b15b05543e5538500b873b0c2c889add6cec90c662cd0da" Oct 31 00:37:54.313385 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.313365 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8e42586c9f22feb9b15b05543e5538500b873b0c2c889add6cec90c662cd0da"} err="failed to get container status \"e8e42586c9f22feb9b15b05543e5538500b873b0c2c889add6cec90c662cd0da\": rpc error: code = NotFound desc = could not find container \"e8e42586c9f22feb9b15b05543e5538500b873b0c2c889add6cec90c662cd0da\": container with ID starting with e8e42586c9f22feb9b15b05543e5538500b873b0c2c889add6cec90c662cd0da not found: ID does not exist" Oct 31 00:37:54.313432 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.313387 5274 scope.go:122] "RemoveContainer" containerID="71fa5393b769366c416449f82bd3143db69a43e18de39b3d7f9c9d831540a48a" Oct 31 00:37:54.313577 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.313559 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71fa5393b769366c416449f82bd3143db69a43e18de39b3d7f9c9d831540a48a"} err="failed to get container status \"71fa5393b769366c416449f82bd3143db69a43e18de39b3d7f9c9d831540a48a\": rpc error: code = NotFound desc = could not find container \"71fa5393b769366c416449f82bd3143db69a43e18de39b3d7f9c9d831540a48a\": container with ID starting with 71fa5393b769366c416449f82bd3143db69a43e18de39b3d7f9c9d831540a48a not found: ID does not exist" Oct 31 00:37:54.313644 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.313578 5274 scope.go:122] "RemoveContainer" containerID="47de2c4275cc951b65c9360d45be4b57ab1bf2ce5a8c29d55731c98b95b5c5e7" Oct 31 00:37:54.313787 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.313770 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47de2c4275cc951b65c9360d45be4b57ab1bf2ce5a8c29d55731c98b95b5c5e7"} err="failed to get container status \"47de2c4275cc951b65c9360d45be4b57ab1bf2ce5a8c29d55731c98b95b5c5e7\": rpc error: code = NotFound desc = could not find container \"47de2c4275cc951b65c9360d45be4b57ab1bf2ce5a8c29d55731c98b95b5c5e7\": container with ID starting with 47de2c4275cc951b65c9360d45be4b57ab1bf2ce5a8c29d55731c98b95b5c5e7 not found: ID does not exist" Oct 31 00:37:54.313835 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.313788 5274 scope.go:122] "RemoveContainer" containerID="2b24cbad42e3d9e26b15f0c49f8bbd9c7f6be7c5ff9b4770d486552f0e95db4a" Oct 31 00:37:54.314036 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.314006 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b24cbad42e3d9e26b15f0c49f8bbd9c7f6be7c5ff9b4770d486552f0e95db4a"} err="failed to get container status \"2b24cbad42e3d9e26b15f0c49f8bbd9c7f6be7c5ff9b4770d486552f0e95db4a\": rpc error: code = NotFound desc = could not find container \"2b24cbad42e3d9e26b15f0c49f8bbd9c7f6be7c5ff9b4770d486552f0e95db4a\": container with ID starting with 2b24cbad42e3d9e26b15f0c49f8bbd9c7f6be7c5ff9b4770d486552f0e95db4a not found: ID does not exist" Oct 31 00:37:54.314036 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.314034 5274 scope.go:122] "RemoveContainer" containerID="e8e42586c9f22feb9b15b05543e5538500b873b0c2c889add6cec90c662cd0da" Oct 31 00:37:54.314276 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.314259 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8e42586c9f22feb9b15b05543e5538500b873b0c2c889add6cec90c662cd0da"} err="failed to get container status \"e8e42586c9f22feb9b15b05543e5538500b873b0c2c889add6cec90c662cd0da\": rpc error: code = NotFound desc = could not find container \"e8e42586c9f22feb9b15b05543e5538500b873b0c2c889add6cec90c662cd0da\": container with ID starting with e8e42586c9f22feb9b15b05543e5538500b873b0c2c889add6cec90c662cd0da not found: ID does not exist" Oct 31 00:37:54.314331 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.314277 5274 scope.go:122] "RemoveContainer" containerID="71fa5393b769366c416449f82bd3143db69a43e18de39b3d7f9c9d831540a48a" Oct 31 00:37:54.314476 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.314460 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71fa5393b769366c416449f82bd3143db69a43e18de39b3d7f9c9d831540a48a"} err="failed to get container status \"71fa5393b769366c416449f82bd3143db69a43e18de39b3d7f9c9d831540a48a\": rpc error: code = NotFound desc = could not find container \"71fa5393b769366c416449f82bd3143db69a43e18de39b3d7f9c9d831540a48a\": container with ID starting with 71fa5393b769366c416449f82bd3143db69a43e18de39b3d7f9c9d831540a48a not found: ID does not exist" Oct 31 00:37:54.314515 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.314477 5274 scope.go:122] "RemoveContainer" containerID="47de2c4275cc951b65c9360d45be4b57ab1bf2ce5a8c29d55731c98b95b5c5e7" Oct 31 00:37:54.314671 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.314646 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47de2c4275cc951b65c9360d45be4b57ab1bf2ce5a8c29d55731c98b95b5c5e7"} err="failed to get container status \"47de2c4275cc951b65c9360d45be4b57ab1bf2ce5a8c29d55731c98b95b5c5e7\": rpc error: code = NotFound desc = could not find container \"47de2c4275cc951b65c9360d45be4b57ab1bf2ce5a8c29d55731c98b95b5c5e7\": container with ID starting with 47de2c4275cc951b65c9360d45be4b57ab1bf2ce5a8c29d55731c98b95b5c5e7 not found: ID does not exist" Oct 31 00:37:54.317393 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.317356 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"638c279f-efbc-45ec-ac21-5d47e6def796\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " Oct 31 00:37:54.317467 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.317456 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"638c279f-efbc-45ec-ac21-5d47e6def796\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " Oct 31 00:37:54.317501 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.317481 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/638c279f-efbc-45ec-ac21-5d47e6def796-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/638c279f-efbc-45ec-ac21-5d47e6def796-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"638c279f-efbc-45ec-ac21-5d47e6def796\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " Oct 31 00:37:54.317532 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.317507 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"638c279f-efbc-45ec-ac21-5d47e6def796\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " Oct 31 00:37:54.317573 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.317532 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/638c279f-efbc-45ec-ac21-5d47e6def796-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/638c279f-efbc-45ec-ac21-5d47e6def796-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"638c279f-efbc-45ec-ac21-5d47e6def796\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " Oct 31 00:37:54.317623 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.317575 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/638c279f-efbc-45ec-ac21-5d47e6def796-tls-assets\" (UniqueName: \"kubernetes.io/projected/638c279f-efbc-45ec-ac21-5d47e6def796-tls-assets\") pod \"638c279f-efbc-45ec-ac21-5d47e6def796\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " Oct 31 00:37:54.317623 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.317593 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/638c279f-efbc-45ec-ac21-5d47e6def796-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/638c279f-efbc-45ec-ac21-5d47e6def796-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"638c279f-efbc-45ec-ac21-5d47e6def796\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " Oct 31 00:37:54.317722 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.317627 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-secret-prometheus-web-tls-ca\") pod \"638c279f-efbc-45ec-ac21-5d47e6def796\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " Oct 31 00:37:54.317722 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.317672 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-config\" (UniqueName: \"kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-config\") pod \"638c279f-efbc-45ec-ac21-5d47e6def796\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " Oct 31 00:37:54.317722 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.317717 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-thanos-prometheus-http-client-file\") pod \"638c279f-efbc-45ec-ac21-5d47e6def796\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " Oct 31 00:37:54.317875 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.317755 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/638c279f-efbc-45ec-ac21-5d47e6def796-kube-api-access-gwnd9\" (UniqueName: \"kubernetes.io/projected/638c279f-efbc-45ec-ac21-5d47e6def796-kube-api-access-gwnd9\") pod \"638c279f-efbc-45ec-ac21-5d47e6def796\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " Oct 31 00:37:54.317875 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.317783 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/638c279f-efbc-45ec-ac21-5d47e6def796-config-out\" (UniqueName: \"kubernetes.io/empty-dir/638c279f-efbc-45ec-ac21-5d47e6def796-config-out\") pod \"638c279f-efbc-45ec-ac21-5d47e6def796\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " Oct 31 00:37:54.317875 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.317838 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-web-config\" (UniqueName: \"kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-web-config\") pod \"638c279f-efbc-45ec-ac21-5d47e6def796\" (UID: \"638c279f-efbc-45ec-ac21-5d47e6def796\") " Oct 31 00:37:54.318039 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.317891 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/638c279f-efbc-45ec-ac21-5d47e6def796-prometheus-data-science-monitoringstack-rulefiles-1" pod "638c279f-efbc-45ec-ac21-5d47e6def796" (UID: "638c279f-efbc-45ec-ac21-5d47e6def796"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:54.318106 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.318059 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/638c279f-efbc-45ec-ac21-5d47e6def796-prometheus-data-science-monitoringstack-rulefiles-2" pod "638c279f-efbc-45ec-ac21-5d47e6def796" (UID: "638c279f-efbc-45ec-ac21-5d47e6def796"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:54.318212 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.318161 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/638c279f-efbc-45ec-ac21-5d47e6def796-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:54.318212 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.318189 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/638c279f-efbc-45ec-ac21-5d47e6def796-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:54.318513 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.318493 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/638c279f-efbc-45ec-ac21-5d47e6def796-prometheus-data-science-monitoringstack-rulefiles-0" pod "638c279f-efbc-45ec-ac21-5d47e6def796" (UID: "638c279f-efbc-45ec-ac21-5d47e6def796"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:54.319880 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.319855 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "638c279f-efbc-45ec-ac21-5d47e6def796" (UID: "638c279f-efbc-45ec-ac21-5d47e6def796"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:54.320420 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.320395 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "638c279f-efbc-45ec-ac21-5d47e6def796" (UID: "638c279f-efbc-45ec-ac21-5d47e6def796"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:54.320420 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.320398 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/638c279f-efbc-45ec-ac21-5d47e6def796-tls-assets" pod "638c279f-efbc-45ec-ac21-5d47e6def796" (UID: "638c279f-efbc-45ec-ac21-5d47e6def796"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:54.320669 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.320652 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-thanos-prometheus-http-client-file" pod "638c279f-efbc-45ec-ac21-5d47e6def796" (UID: "638c279f-efbc-45ec-ac21-5d47e6def796"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:54.321388 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.321368 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/638c279f-efbc-45ec-ac21-5d47e6def796-config-out" pod "638c279f-efbc-45ec-ac21-5d47e6def796" (UID: "638c279f-efbc-45ec-ac21-5d47e6def796"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:37:54.321894 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.321873 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-secret-prometheus-web-tls-ca" pod "638c279f-efbc-45ec-ac21-5d47e6def796" (UID: "638c279f-efbc-45ec-ac21-5d47e6def796"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:54.322256 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.322235 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/638c279f-efbc-45ec-ac21-5d47e6def796-kube-api-access-gwnd9" pod "638c279f-efbc-45ec-ac21-5d47e6def796" (UID: "638c279f-efbc-45ec-ac21-5d47e6def796"). InnerVolumeSpecName "kube-api-access-gwnd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:54.322630 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.322611 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-config" pod "638c279f-efbc-45ec-ac21-5d47e6def796" (UID: "638c279f-efbc-45ec-ac21-5d47e6def796"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:54.322708 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.322652 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1" pod "638c279f-efbc-45ec-ac21-5d47e6def796" (UID: "638c279f-efbc-45ec-ac21-5d47e6def796"). InnerVolumeSpecName "pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:37:54.331782 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.331762 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-web-config" pod "638c279f-efbc-45ec-ac21-5d47e6def796" (UID: "638c279f-efbc-45ec-ac21-5d47e6def796"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:54.419524 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.419497 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwnd9\" (UniqueName: \"kubernetes.io/projected/638c279f-efbc-45ec-ac21-5d47e6def796-kube-api-access-gwnd9\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:54.419524 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.419520 5274 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/638c279f-efbc-45ec-ac21-5d47e6def796-config-out\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:54.419699 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.419531 5274 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-web-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:54.419699 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:37:54.419543 5274 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-collector-collector-a0f9182a: configmap "data-science-collector-collector-a0f9182a" not found Oct 31 00:37:54.419699 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:37:54.419555 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-tls: secret "data-science-collector-tls" not found Oct 31 00:37:54.419699 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:37:54.419558 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found Oct 31 00:37:54.419699 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.419546 5274 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:54.419699 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:37:54.419601 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-monitoring-tls: secret "data-science-collector-monitoring-tls" not found Oct 31 00:37:54.419699 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:37:54.419616 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-otc-internal podName:7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8 nodeName:}" failed. No retries permitted until 2026-10-31 00:37:54.919594658 +0000 UTC m=+2311.556413372 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otc-internal" (UniqueName: "kubernetes.io/configmap/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-otc-internal") pod "data-science-collector-collector-0" (UID: "7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8") : configmap "data-science-collector-collector-a0f9182a" not found Oct 31 00:37:54.419699 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:37:54.419660 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-tls-certs podName:7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8 nodeName:}" failed. No retries permitted until 2026-10-31 00:37:54.919646065 +0000 UTC m=+2311.556464760 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-tls-certs") pod "data-science-collector-collector-0" (UID: "7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8") : secret "data-science-collector-tls" not found Oct 31 00:37:54.419699 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:37:54.419676 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-otlp-tls-certs podName:7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8 nodeName:}" failed. No retries permitted until 2026-10-31 00:37:54.919668149 +0000 UTC m=+2311.556486845 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-otlp-tls-certs") pod "data-science-collector-collector-0" (UID: "7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8") : secret "data-science-collector-collector-headless-tls" not found Oct 31 00:37:54.419699 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:37:54.419690 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-monitoring-tls-certs podName:7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8 nodeName:}" failed. No retries permitted until 2026-10-31 00:37:54.919682772 +0000 UTC m=+2311.556501467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-tls-certs" (UniqueName: "kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-monitoring-tls-certs") pod "data-science-collector-collector-0" (UID: "7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8") : secret "data-science-collector-monitoring-tls" not found Oct 31 00:37:54.419699 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.419707 5274 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" " Oct 31 00:37:54.420278 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.419718 5274 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:54.420278 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.419730 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/638c279f-efbc-45ec-ac21-5d47e6def796-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:54.420278 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.419739 5274 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/638c279f-efbc-45ec-ac21-5d47e6def796-tls-assets\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:54.420278 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.419748 5274 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-secret-prometheus-web-tls-ca\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:54.420278 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.419756 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:54.420278 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.419764 5274 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/638c279f-efbc-45ec-ac21-5d47e6def796-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:54.430789 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.430774 5274 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1") on node "ip-10-0-129-70.us-east-2.compute.internal" Oct 31 00:37:54.520996 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.520976 5274 reconciler_common.go:299] "Volume detached for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"csi-51e8027f163b5f98040faf3f8c30848dd44a7a90ca2bd0af2446ea6398ed018d\"" Oct 31 00:37:54.588327 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.588302 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:37:54.592660 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:54.592639 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:37:54.923819 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:37:54.923790 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-monitoring-tls: secret "data-science-collector-monitoring-tls" not found Oct 31 00:37:54.923819 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:37:54.923816 5274 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-collector-collector-a0f9182a: configmap "data-science-collector-collector-a0f9182a" not found Oct 31 00:37:54.924019 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:37:54.923795 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-tls: secret "data-science-collector-tls" not found Oct 31 00:37:54.924019 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:37:54.923793 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found Oct 31 00:37:54.924019 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:37:54.923861 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-monitoring-tls-certs podName:7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8 nodeName:}" failed. No retries permitted until 2026-10-31 00:37:55.923848786 +0000 UTC m=+2312.560667480 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "monitoring-tls-certs" (UniqueName: "kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-monitoring-tls-certs") pod "data-science-collector-collector-0" (UID: "7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8") : secret "data-science-collector-monitoring-tls" not found Oct 31 00:37:54.924019 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:37:54.923935 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-otc-internal podName:7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8 nodeName:}" failed. No retries permitted until 2026-10-31 00:37:55.923914648 +0000 UTC m=+2312.560733349 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "otc-internal" (UniqueName: "kubernetes.io/configmap/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-otc-internal") pod "data-science-collector-collector-0" (UID: "7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8") : configmap "data-science-collector-collector-a0f9182a" not found Oct 31 00:37:54.924019 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:37:54.923971 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-tls-certs podName:7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8 nodeName:}" failed. No retries permitted until 2026-10-31 00:37:55.92394453 +0000 UTC m=+2312.560763224 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-tls-certs") pod "data-science-collector-collector-0" (UID: "7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8") : secret "data-science-collector-tls" not found Oct 31 00:37:54.924019 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:37:54.923986 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-otlp-tls-certs podName:7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8 nodeName:}" failed. No retries permitted until 2026-10-31 00:37:55.923977655 +0000 UTC m=+2312.560796355 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-otlp-tls-certs") pod "data-science-collector-collector-0" (UID: "7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8") : secret "data-science-collector-collector-headless-tls" not found Oct 31 00:37:55.272173 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:55.272100 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8" containerName="otc-container" containerID="cri-o://7a0b85fedb4f4506a47c880b497e9dd62394da4054f5f2108379f20378dd6431" gracePeriod=30 Oct 31 00:37:55.481463 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:55.481430 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dc6465e-bbd3-4e25-b4c6-d28231ef86d6" path="/var/lib/kubelet/pods/3dc6465e-bbd3-4e25-b4c6-d28231ef86d6/volumes" Oct 31 00:37:55.481812 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:55.481799 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53262348-aaeb-4648-8b93-3b4c40f3ebc2" path="/var/lib/kubelet/pods/53262348-aaeb-4648-8b93-3b4c40f3ebc2/volumes" Oct 31 00:37:55.482257 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:55.482244 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="638c279f-efbc-45ec-ac21-5d47e6def796" path="/var/lib/kubelet/pods/638c279f-efbc-45ec-ac21-5d47e6def796/volumes" Oct 31 00:37:55.518144 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:55.518122 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:55.628628 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:55.628606 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-kube-api-access-pql28\" (UniqueName: \"kubernetes.io/projected/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-kube-api-access-pql28\") pod \"7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8\" (UID: \"7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8\") " Oct 31 00:37:55.628769 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:55.628661 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-tls-certs\" (UniqueName: \"kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-tls-certs\") pod \"7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8\" (UID: \"7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8\") " Oct 31 00:37:55.628769 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:55.628727 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-otc-internal\" (UniqueName: \"kubernetes.io/configmap/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-otc-internal\") pod \"7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8\" (UID: \"7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8\") " Oct 31 00:37:55.628875 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:55.628767 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-monitoring-tls-certs\") pod \"7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8\" (UID: \"7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8\") " Oct 31 00:37:55.628875 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:55.628808 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-data-science-collector-ta-client-cert\") pod \"7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8\" (UID: \"7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8\") " Oct 31 00:37:55.628875 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:55.628860 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-otlp-tls-certs\") pod \"7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8\" (UID: \"7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8\") " Oct 31 00:37:55.629454 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:55.629149 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-otc-internal" pod "7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8" (UID: "7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:55.631192 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:55.631169 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-otlp-tls-certs" pod "7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8" (UID: "7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:55.631436 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:55.631416 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-data-science-collector-ta-client-cert" pod "7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8" (UID: "7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:55.631490 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:55.631444 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-kube-api-access-pql28" pod "7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8" (UID: "7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8"). InnerVolumeSpecName "kube-api-access-pql28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:55.631490 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:55.631472 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-monitoring-tls-certs" pod "7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8" (UID: "7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:55.631590 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:55.631575 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-tls-certs" pod "7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8" (UID: "7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:55.729740 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:55.729712 5274 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-data-science-collector-ta-client-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:55.729740 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:55.729737 5274 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-otlp-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:55.729867 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:55.729755 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pql28\" (UniqueName: \"kubernetes.io/projected/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-kube-api-access-pql28\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:55.729867 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:55.729768 5274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:55.729867 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:55.729782 5274 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-otc-internal\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:55.729867 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:55.729794 5274 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8-monitoring-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:56.279076 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:56.279041 5274 generic.go:356] "Generic (PLEG): container finished" podID="7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8" containerID="7a0b85fedb4f4506a47c880b497e9dd62394da4054f5f2108379f20378dd6431" exitCode=0 Oct 31 00:37:56.279744 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:56.279085 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:37:56.279744 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:56.279128 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8","Type":"ContainerDied","Data":"7a0b85fedb4f4506a47c880b497e9dd62394da4054f5f2108379f20378dd6431"} Oct 31 00:37:56.279744 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:56.279178 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8","Type":"ContainerDied","Data":"a2e75b53c747fb5930801931a249d6ec607e2b24d86f17d2fd386f509b772e89"} Oct 31 00:37:56.279744 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:56.279199 5274 scope.go:122] "RemoveContainer" containerID="7a0b85fedb4f4506a47c880b497e9dd62394da4054f5f2108379f20378dd6431" Oct 31 00:37:56.288339 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:56.288317 5274 scope.go:122] "RemoveContainer" containerID="7a0b85fedb4f4506a47c880b497e9dd62394da4054f5f2108379f20378dd6431" Oct 31 00:37:56.288598 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:37:56.288579 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a0b85fedb4f4506a47c880b497e9dd62394da4054f5f2108379f20378dd6431\": container with ID starting with 7a0b85fedb4f4506a47c880b497e9dd62394da4054f5f2108379f20378dd6431 not found: ID does not exist" containerID="7a0b85fedb4f4506a47c880b497e9dd62394da4054f5f2108379f20378dd6431" Oct 31 00:37:56.288674 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:56.288604 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a0b85fedb4f4506a47c880b497e9dd62394da4054f5f2108379f20378dd6431"} err="failed to get container status \"7a0b85fedb4f4506a47c880b497e9dd62394da4054f5f2108379f20378dd6431\": rpc error: code = NotFound desc = could not find container \"7a0b85fedb4f4506a47c880b497e9dd62394da4054f5f2108379f20378dd6431\": container with ID starting with 7a0b85fedb4f4506a47c880b497e9dd62394da4054f5f2108379f20378dd6431 not found: ID does not exist" Oct 31 00:37:56.300565 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:56.300544 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:37:56.305037 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:56.305018 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:37:57.480362 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:57.480331 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8" path="/var/lib/kubelet/pods/7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8/volumes" Oct 31 00:37:57.667730 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:37:57.667692 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="1294ed53-4e5a-4105-8acd-b2a04cce8df9" containerName="alertmanager" probeResult="failure" output="Get \"http://10.134.0.89:9093/-/ready\": dial tcp 10.134.0.89:9093: connect: connection refused" Oct 31 00:38:01.033081 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.033051 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jrq79"] Oct 31 00:38:01.033440 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.033377 5274 cpu_manager.go:414] "containerMap: removing container" podUID="3dc6465e-bbd3-4e25-b4c6-d28231ef86d6" containerName="kube-rbac-proxy" Oct 31 00:38:01.033440 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.033392 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dc6465e-bbd3-4e25-b4c6-d28231ef86d6" containerName="kube-rbac-proxy" Oct 31 00:38:01.033440 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.033426 5274 cpu_manager.go:414] "containerMap: removing container" podUID="638c279f-efbc-45ec-ac21-5d47e6def796" containerName="prometheus" Oct 31 00:38:01.033440 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.033437 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="638c279f-efbc-45ec-ac21-5d47e6def796" containerName="prometheus" Oct 31 00:38:01.033577 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.033450 5274 cpu_manager.go:414] "containerMap: removing container" podUID="638c279f-efbc-45ec-ac21-5d47e6def796" containerName="thanos-sidecar" Oct 31 00:38:01.033577 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.033457 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="638c279f-efbc-45ec-ac21-5d47e6def796" containerName="thanos-sidecar" Oct 31 00:38:01.033577 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.033470 5274 cpu_manager.go:414] "containerMap: removing container" podUID="638c279f-efbc-45ec-ac21-5d47e6def796" containerName="config-reloader" Oct 31 00:38:01.033577 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.033478 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="638c279f-efbc-45ec-ac21-5d47e6def796" containerName="config-reloader" Oct 31 00:38:01.033577 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.033490 5274 cpu_manager.go:414] "containerMap: removing container" podUID="53262348-aaeb-4648-8b93-3b4c40f3ebc2" containerName="korrel8r" Oct 31 00:38:01.033577 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.033497 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53262348-aaeb-4648-8b93-3b4c40f3ebc2" containerName="korrel8r" Oct 31 00:38:01.033577 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.033507 5274 cpu_manager.go:414] "containerMap: removing container" podUID="7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8" containerName="otc-container" Oct 31 00:38:01.033577 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.033515 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8" containerName="otc-container" Oct 31 00:38:01.033577 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.033553 5274 cpu_manager.go:414] "containerMap: removing container" podUID="638c279f-efbc-45ec-ac21-5d47e6def796" containerName="init-config-reloader" Oct 31 00:38:01.033577 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.033560 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="638c279f-efbc-45ec-ac21-5d47e6def796" containerName="init-config-reloader" Oct 31 00:38:01.034051 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.033644 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dc6465e-bbd3-4e25-b4c6-d28231ef86d6" containerName="kube-rbac-proxy" Oct 31 00:38:01.034051 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.033693 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="638c279f-efbc-45ec-ac21-5d47e6def796" containerName="prometheus" Oct 31 00:38:01.034051 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.033705 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="638c279f-efbc-45ec-ac21-5d47e6def796" containerName="thanos-sidecar" Oct 31 00:38:01.034051 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.033716 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="638c279f-efbc-45ec-ac21-5d47e6def796" containerName="config-reloader" Oct 31 00:38:01.034051 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.033724 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="53262348-aaeb-4648-8b93-3b4c40f3ebc2" containerName="korrel8r" Oct 31 00:38:01.034051 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.033732 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cf5b299-5a4a-45b2-b8dc-a1124ee37cb8" containerName="otc-container" Oct 31 00:38:01.038301 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.038278 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jrq79" Oct 31 00:38:01.045844 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.045820 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:38:01.045999 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.045886 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 31 00:38:01.045999 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.045908 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\"" Oct 31 00:38:01.045999 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.045992 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\"" Oct 31 00:38:01.047196 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.047171 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-h2s9k\"" Oct 31 00:38:01.047906 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.047884 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jrq79"] Oct 31 00:38:01.094873 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.094844 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-9c4wg"] Oct 31 00:38:01.098165 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.098148 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-9c4wg" Oct 31 00:38:01.109593 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.109569 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-9c4wg"] Oct 31 00:38:01.145787 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.145763 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 31 00:38:01.146136 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.146120 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 31 00:38:01.146179 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.146135 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-kcwg6\"" Oct 31 00:38:01.171479 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.171456 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1db65176-7e26-48d5-b66d-ea279099c362-tls\") pod \"korrel8r-5cbcc89b8f-9c4wg\" (UID: \"1db65176-7e26-48d5-b66d-ea279099c362\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-9c4wg" Oct 31 00:38:01.171553 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.171498 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/37c7d1c2-01c8-47f3-8e6c-7b14918b6743-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-jrq79\" (UID: \"37c7d1c2-01c8-47f3-8e6c-7b14918b6743\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jrq79" Oct 31 00:38:01.171690 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.171576 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/37c7d1c2-01c8-47f3-8e6c-7b14918b6743-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-jrq79\" (UID: \"37c7d1c2-01c8-47f3-8e6c-7b14918b6743\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jrq79" Oct 31 00:38:01.171690 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.171608 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/37c7d1c2-01c8-47f3-8e6c-7b14918b6743-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-jrq79\" (UID: \"37c7d1c2-01c8-47f3-8e6c-7b14918b6743\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jrq79" Oct 31 00:38:01.171690 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.171629 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhf5v\" (UniqueName: \"kubernetes.io/projected/37c7d1c2-01c8-47f3-8e6c-7b14918b6743-kube-api-access-lhf5v\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-jrq79\" (UID: \"37c7d1c2-01c8-47f3-8e6c-7b14918b6743\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jrq79" Oct 31 00:38:01.171690 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.171651 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37c7d1c2-01c8-47f3-8e6c-7b14918b6743-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-jrq79\" (UID: \"37c7d1c2-01c8-47f3-8e6c-7b14918b6743\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jrq79" Oct 31 00:38:01.171811 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.171751 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1db65176-7e26-48d5-b66d-ea279099c362-config\") pod \"korrel8r-5cbcc89b8f-9c4wg\" (UID: \"1db65176-7e26-48d5-b66d-ea279099c362\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-9c4wg" Oct 31 00:38:01.171811 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.171804 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkhzl\" (UniqueName: \"kubernetes.io/projected/1db65176-7e26-48d5-b66d-ea279099c362-kube-api-access-hkhzl\") pod \"korrel8r-5cbcc89b8f-9c4wg\" (UID: \"1db65176-7e26-48d5-b66d-ea279099c362\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-9c4wg" Oct 31 00:38:01.272495 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.272461 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1db65176-7e26-48d5-b66d-ea279099c362-tls\") pod \"korrel8r-5cbcc89b8f-9c4wg\" (UID: \"1db65176-7e26-48d5-b66d-ea279099c362\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-9c4wg" Oct 31 00:38:01.272650 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.272516 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/37c7d1c2-01c8-47f3-8e6c-7b14918b6743-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-jrq79\" (UID: \"37c7d1c2-01c8-47f3-8e6c-7b14918b6743\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jrq79" Oct 31 00:38:01.272650 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.272545 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/37c7d1c2-01c8-47f3-8e6c-7b14918b6743-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-jrq79\" (UID: \"37c7d1c2-01c8-47f3-8e6c-7b14918b6743\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jrq79" Oct 31 00:38:01.272650 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.272573 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/37c7d1c2-01c8-47f3-8e6c-7b14918b6743-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-jrq79\" (UID: \"37c7d1c2-01c8-47f3-8e6c-7b14918b6743\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jrq79" Oct 31 00:38:01.272650 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.272589 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhf5v\" (UniqueName: \"kubernetes.io/projected/37c7d1c2-01c8-47f3-8e6c-7b14918b6743-kube-api-access-lhf5v\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-jrq79\" (UID: \"37c7d1c2-01c8-47f3-8e6c-7b14918b6743\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jrq79" Oct 31 00:38:01.272650 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.272608 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37c7d1c2-01c8-47f3-8e6c-7b14918b6743-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-jrq79\" (UID: \"37c7d1c2-01c8-47f3-8e6c-7b14918b6743\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jrq79" Oct 31 00:38:01.272650 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:38:01.272635 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/korrel8r-tls: secret "korrel8r-tls" not found Oct 31 00:38:01.272983 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:38:01.272712 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1db65176-7e26-48d5-b66d-ea279099c362-tls podName:1db65176-7e26-48d5-b66d-ea279099c362 nodeName:}" failed. No retries permitted until 2026-10-31 00:38:01.772690333 +0000 UTC m=+2318.409509051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls" (UniqueName: "kubernetes.io/secret/1db65176-7e26-48d5-b66d-ea279099c362-tls") pod "korrel8r-5cbcc89b8f-9c4wg" (UID: "1db65176-7e26-48d5-b66d-ea279099c362") : secret "korrel8r-tls" not found Oct 31 00:38:01.273062 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.272638 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1db65176-7e26-48d5-b66d-ea279099c362-config\") pod \"korrel8r-5cbcc89b8f-9c4wg\" (UID: \"1db65176-7e26-48d5-b66d-ea279099c362\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-9c4wg" Oct 31 00:38:01.273119 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.273066 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkhzl\" (UniqueName: \"kubernetes.io/projected/1db65176-7e26-48d5-b66d-ea279099c362-kube-api-access-hkhzl\") pod \"korrel8r-5cbcc89b8f-9c4wg\" (UID: \"1db65176-7e26-48d5-b66d-ea279099c362\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-9c4wg" Oct 31 00:38:01.273475 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:38:01.273456 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-prometheus-cluster-proxy-tls: secret "data-science-prometheus-cluster-proxy-tls" not found Oct 31 00:38:01.273549 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:38:01.273514 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37c7d1c2-01c8-47f3-8e6c-7b14918b6743-tls podName:37c7d1c2-01c8-47f3-8e6c-7b14918b6743 nodeName:}" failed. No retries permitted until 2026-10-31 00:38:01.773497974 +0000 UTC m=+2318.410316684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls" (UniqueName: "kubernetes.io/secret/37c7d1c2-01c8-47f3-8e6c-7b14918b6743-tls") pod "data-science-prometheus-cluster-proxy-545f759f7c-jrq79" (UID: "37c7d1c2-01c8-47f3-8e6c-7b14918b6743") : secret "data-science-prometheus-cluster-proxy-tls" not found Oct 31 00:38:01.273611 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.273591 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1db65176-7e26-48d5-b66d-ea279099c362-config\") pod \"korrel8r-5cbcc89b8f-9c4wg\" (UID: \"1db65176-7e26-48d5-b66d-ea279099c362\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-9c4wg" Oct 31 00:38:01.274352 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.274207 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/37c7d1c2-01c8-47f3-8e6c-7b14918b6743-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-jrq79\" (UID: \"37c7d1c2-01c8-47f3-8e6c-7b14918b6743\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jrq79" Oct 31 00:38:01.276268 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.276249 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/37c7d1c2-01c8-47f3-8e6c-7b14918b6743-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-jrq79\" (UID: \"37c7d1c2-01c8-47f3-8e6c-7b14918b6743\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jrq79" Oct 31 00:38:01.276352 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.276269 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37c7d1c2-01c8-47f3-8e6c-7b14918b6743-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-jrq79\" (UID: \"37c7d1c2-01c8-47f3-8e6c-7b14918b6743\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jrq79" Oct 31 00:38:01.285496 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.285449 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkhzl\" (UniqueName: \"kubernetes.io/projected/1db65176-7e26-48d5-b66d-ea279099c362-kube-api-access-hkhzl\") pod \"korrel8r-5cbcc89b8f-9c4wg\" (UID: \"1db65176-7e26-48d5-b66d-ea279099c362\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-9c4wg" Oct 31 00:38:01.286432 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.286409 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhf5v\" (UniqueName: \"kubernetes.io/projected/37c7d1c2-01c8-47f3-8e6c-7b14918b6743-kube-api-access-lhf5v\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-jrq79\" (UID: \"37c7d1c2-01c8-47f3-8e6c-7b14918b6743\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jrq79" Oct 31 00:38:01.532223 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.532192 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:38:01.537248 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.537194 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:38:01.543712 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.543689 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:38:01.546477 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.546451 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:38:01.547557 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.547536 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:38:01.547715 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.547608 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:38:01.547836 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.547820 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:38:01.547940 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.547926 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:38:01.678352 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.678296 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ed80a10-b519-4d87-a4a4-0783c84af9aa-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5ed80a10-b519-4d87-a4a4-0783c84af9aa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:38:01.678512 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.678413 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ed80a10-b519-4d87-a4a4-0783c84af9aa-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5ed80a10-b519-4d87-a4a4-0783c84af9aa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:38:01.678512 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.678458 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5ed80a10-b519-4d87-a4a4-0783c84af9aa-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5ed80a10-b519-4d87-a4a4-0783c84af9aa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:38:01.678512 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.678476 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5ed80a10-b519-4d87-a4a4-0783c84af9aa-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5ed80a10-b519-4d87-a4a4-0783c84af9aa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:38:01.678616 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.678602 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cfsd\" (UniqueName: \"kubernetes.io/projected/5ed80a10-b519-4d87-a4a4-0783c84af9aa-kube-api-access-8cfsd\") pod \"data-science-collector-collector-0\" (UID: \"5ed80a10-b519-4d87-a4a4-0783c84af9aa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:38:01.678677 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.678665 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ed80a10-b519-4d87-a4a4-0783c84af9aa-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5ed80a10-b519-4d87-a4a4-0783c84af9aa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:38:01.779400 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.779282 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1db65176-7e26-48d5-b66d-ea279099c362-tls\") pod \"korrel8r-5cbcc89b8f-9c4wg\" (UID: \"1db65176-7e26-48d5-b66d-ea279099c362\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-9c4wg" Oct 31 00:38:01.780014 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.779978 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ed80a10-b519-4d87-a4a4-0783c84af9aa-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5ed80a10-b519-4d87-a4a4-0783c84af9aa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:38:01.780149 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.780064 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ed80a10-b519-4d87-a4a4-0783c84af9aa-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5ed80a10-b519-4d87-a4a4-0783c84af9aa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:38:01.780149 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.780105 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/37c7d1c2-01c8-47f3-8e6c-7b14918b6743-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-jrq79\" (UID: \"37c7d1c2-01c8-47f3-8e6c-7b14918b6743\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jrq79" Oct 31 00:38:01.780149 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.780139 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ed80a10-b519-4d87-a4a4-0783c84af9aa-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5ed80a10-b519-4d87-a4a4-0783c84af9aa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:38:01.780339 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.780178 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5ed80a10-b519-4d87-a4a4-0783c84af9aa-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5ed80a10-b519-4d87-a4a4-0783c84af9aa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:38:01.780339 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.780205 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5ed80a10-b519-4d87-a4a4-0783c84af9aa-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5ed80a10-b519-4d87-a4a4-0783c84af9aa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:38:01.780339 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:38:01.780234 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found Oct 31 00:38:01.780339 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.780258 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cfsd\" (UniqueName: \"kubernetes.io/projected/5ed80a10-b519-4d87-a4a4-0783c84af9aa-kube-api-access-8cfsd\") pod \"data-science-collector-collector-0\" (UID: \"5ed80a10-b519-4d87-a4a4-0783c84af9aa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:38:01.780339 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:38:01.780307 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ed80a10-b519-4d87-a4a4-0783c84af9aa-otlp-tls-certs podName:5ed80a10-b519-4d87-a4a4-0783c84af9aa nodeName:}" failed. No retries permitted until 2026-10-31 00:38:02.280287644 +0000 UTC m=+2318.917106371 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/5ed80a10-b519-4d87-a4a4-0783c84af9aa-otlp-tls-certs") pod "data-science-collector-collector-0" (UID: "5ed80a10-b519-4d87-a4a4-0783c84af9aa") : secret "data-science-collector-collector-headless-tls" not found Oct 31 00:38:01.781317 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.781286 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5ed80a10-b519-4d87-a4a4-0783c84af9aa-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5ed80a10-b519-4d87-a4a4-0783c84af9aa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:38:01.782656 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.782615 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1db65176-7e26-48d5-b66d-ea279099c362-tls\") pod \"korrel8r-5cbcc89b8f-9c4wg\" (UID: \"1db65176-7e26-48d5-b66d-ea279099c362\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-9c4wg" Oct 31 00:38:01.782842 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.782819 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ed80a10-b519-4d87-a4a4-0783c84af9aa-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5ed80a10-b519-4d87-a4a4-0783c84af9aa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:38:01.783509 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.783403 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5ed80a10-b519-4d87-a4a4-0783c84af9aa-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5ed80a10-b519-4d87-a4a4-0783c84af9aa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:38:01.783881 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.783857 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ed80a10-b519-4d87-a4a4-0783c84af9aa-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5ed80a10-b519-4d87-a4a4-0783c84af9aa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:38:01.785103 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.785084 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/37c7d1c2-01c8-47f3-8e6c-7b14918b6743-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-jrq79\" (UID: \"37c7d1c2-01c8-47f3-8e6c-7b14918b6743\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jrq79" Oct 31 00:38:01.794843 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.794787 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cfsd\" (UniqueName: \"kubernetes.io/projected/5ed80a10-b519-4d87-a4a4-0783c84af9aa-kube-api-access-8cfsd\") pod \"data-science-collector-collector-0\" (UID: \"5ed80a10-b519-4d87-a4a4-0783c84af9aa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:38:01.953537 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:01.953503 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jrq79" Oct 31 00:38:02.010155 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:02.010124 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-9c4wg" Oct 31 00:38:02.227830 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:02.227803 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jrq79"] Oct 31 00:38:02.230269 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:38:02.230244 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37c7d1c2_01c8_47f3_8e6c_7b14918b6743.slice/crio-d18e4dbfb10cbd5e757d8b87d15603abfd816f08f7b9d1657ced2b30562ead2e WatchSource:0}: Error finding container d18e4dbfb10cbd5e757d8b87d15603abfd816f08f7b9d1657ced2b30562ead2e: Status 404 returned error can't find the container with id d18e4dbfb10cbd5e757d8b87d15603abfd816f08f7b9d1657ced2b30562ead2e Oct 31 00:38:02.286029 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:02.286005 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ed80a10-b519-4d87-a4a4-0783c84af9aa-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5ed80a10-b519-4d87-a4a4-0783c84af9aa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:38:02.289042 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:02.289017 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ed80a10-b519-4d87-a4a4-0783c84af9aa-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5ed80a10-b519-4d87-a4a4-0783c84af9aa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:38:02.290288 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:02.290265 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-9c4wg"] Oct 31 00:38:02.291650 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:38:02.291626 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db65176_7e26_48d5_b66d_ea279099c362.slice/crio-d3cb65881ee763a86df1ee51e962d614c740f9d799048d9df1636cf7e6a08a36 WatchSource:0}: Error finding container d3cb65881ee763a86df1ee51e962d614c740f9d799048d9df1636cf7e6a08a36: Status 404 returned error can't find the container with id d3cb65881ee763a86df1ee51e962d614c740f9d799048d9df1636cf7e6a08a36 Oct 31 00:38:02.315339 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:02.315276 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-9c4wg" event={"ID":"1db65176-7e26-48d5-b66d-ea279099c362","Type":"ContainerStarted","Data":"d3cb65881ee763a86df1ee51e962d614c740f9d799048d9df1636cf7e6a08a36"} Oct 31 00:38:02.316477 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:02.316456 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jrq79" event={"ID":"37c7d1c2-01c8-47f3-8e6c-7b14918b6743","Type":"ContainerStarted","Data":"d18e4dbfb10cbd5e757d8b87d15603abfd816f08f7b9d1657ced2b30562ead2e"} Oct 31 00:38:02.458903 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:02.458873 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:38:02.667318 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:02.667283 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="1294ed53-4e5a-4105-8acd-b2a04cce8df9" containerName="alertmanager" probeResult="failure" output="Get \"http://10.134.0.89:9093/-/ready\": dial tcp 10.134.0.89:9093: connect: connection refused" Oct 31 00:38:02.735563 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:02.735540 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:38:02.737568 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:38:02.737537 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ed80a10_b519_4d87_a4a4_0783c84af9aa.slice/crio-bcfad1db1d5ecd5bec24310dad6d9074c83f16c64000dfc4c7b67bf940b67731 WatchSource:0}: Error finding container bcfad1db1d5ecd5bec24310dad6d9074c83f16c64000dfc4c7b67bf940b67731: Status 404 returned error can't find the container with id bcfad1db1d5ecd5bec24310dad6d9074c83f16c64000dfc4c7b67bf940b67731 Oct 31 00:38:02.739727 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:02.739695 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:38:02.739805 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:02.739760 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:38:03.323165 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:03.323129 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-9c4wg" event={"ID":"1db65176-7e26-48d5-b66d-ea279099c362","Type":"ContainerStarted","Data":"9f420f1318939dfee379bbb9881ed3978aa5d3410e7ceab2f19026fc21b0959a"} Oct 31 00:38:03.323607 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:03.323241 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-9c4wg" Oct 31 00:38:03.324853 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:03.324834 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5ed80a10-b519-4d87-a4a4-0783c84af9aa","Type":"ContainerStarted","Data":"c5089e368c3cd83016b26e69d547360b47bfc6a4fe4fb0048916e6c9a9bdd325"} Oct 31 00:38:03.324978 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:03.324874 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5ed80a10-b519-4d87-a4a4-0783c84af9aa","Type":"ContainerStarted","Data":"bcfad1db1d5ecd5bec24310dad6d9074c83f16c64000dfc4c7b67bf940b67731"} Oct 31 00:38:03.326427 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:03.326408 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jrq79" event={"ID":"37c7d1c2-01c8-47f3-8e6c-7b14918b6743","Type":"ContainerStarted","Data":"4fd48a1469c39f8446954a1fb2d2408b628676ff83dcbfce418b835c6967278f"} Oct 31 00:38:03.326494 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:03.326477 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jrq79" Oct 31 00:38:03.342348 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:03.342286 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-9c4wg" podStartSLOduration=2.342269977 podStartE2EDuration="2.342269977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:03.341901098 +0000 UTC m=+2319.978719814" watchObservedRunningTime="2026-10-31 00:38:03.342269977 +0000 UTC m=+2319.979088728" Oct 31 00:38:03.363331 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:03.363279 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jrq79" podStartSLOduration=2.363268134 podStartE2EDuration="2.363268134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:03.362980891 +0000 UTC m=+2319.999799608" watchObservedRunningTime="2026-10-31 00:38:03.363268134 +0000 UTC m=+2320.000086850" Oct 31 00:38:03.388694 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:03.388656 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.38864307 podStartE2EDuration="2.38864307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:03.387294475 +0000 UTC m=+2320.024113191" watchObservedRunningTime="2026-10-31 00:38:03.38864307 +0000 UTC m=+2320.025461803" Oct 31 00:38:03.687311 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:03.687284 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:38:03.797322 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:03.797280 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1294ed53-4e5a-4105-8acd-b2a04cce8df9-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1294ed53-4e5a-4105-8acd-b2a04cce8df9-cluster-tls-config\") pod \"1294ed53-4e5a-4105-8acd-b2a04cce8df9\" (UID: \"1294ed53-4e5a-4105-8acd-b2a04cce8df9\") " Oct 31 00:38:03.797488 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:03.797333 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1294ed53-4e5a-4105-8acd-b2a04cce8df9-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/1294ed53-4e5a-4105-8acd-b2a04cce8df9-alertmanager-data-science-monitoringstack-db\") pod \"1294ed53-4e5a-4105-8acd-b2a04cce8df9\" (UID: \"1294ed53-4e5a-4105-8acd-b2a04cce8df9\") " Oct 31 00:38:03.797488 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:03.797388 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1294ed53-4e5a-4105-8acd-b2a04cce8df9-kube-api-access-b7jqx\" (UniqueName: \"kubernetes.io/projected/1294ed53-4e5a-4105-8acd-b2a04cce8df9-kube-api-access-b7jqx\") pod \"1294ed53-4e5a-4105-8acd-b2a04cce8df9\" (UID: \"1294ed53-4e5a-4105-8acd-b2a04cce8df9\") " Oct 31 00:38:03.797488 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:03.797434 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1294ed53-4e5a-4105-8acd-b2a04cce8df9-web-config\" (UniqueName: \"kubernetes.io/secret/1294ed53-4e5a-4105-8acd-b2a04cce8df9-web-config\") pod \"1294ed53-4e5a-4105-8acd-b2a04cce8df9\" (UID: \"1294ed53-4e5a-4105-8acd-b2a04cce8df9\") " Oct 31 00:38:03.797660 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:03.797497 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1294ed53-4e5a-4105-8acd-b2a04cce8df9-config-out\" (UniqueName: \"kubernetes.io/empty-dir/1294ed53-4e5a-4105-8acd-b2a04cce8df9-config-out\") pod \"1294ed53-4e5a-4105-8acd-b2a04cce8df9\" (UID: \"1294ed53-4e5a-4105-8acd-b2a04cce8df9\") " Oct 31 00:38:03.797660 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:03.797535 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1294ed53-4e5a-4105-8acd-b2a04cce8df9-tls-assets\" (UniqueName: \"kubernetes.io/projected/1294ed53-4e5a-4105-8acd-b2a04cce8df9-tls-assets\") pod \"1294ed53-4e5a-4105-8acd-b2a04cce8df9\" (UID: \"1294ed53-4e5a-4105-8acd-b2a04cce8df9\") " Oct 31 00:38:03.797660 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:03.797586 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1294ed53-4e5a-4105-8acd-b2a04cce8df9-config-volume\" (UniqueName: \"kubernetes.io/secret/1294ed53-4e5a-4105-8acd-b2a04cce8df9-config-volume\") pod \"1294ed53-4e5a-4105-8acd-b2a04cce8df9\" (UID: \"1294ed53-4e5a-4105-8acd-b2a04cce8df9\") " Oct 31 00:38:03.797808 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:03.797739 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1294ed53-4e5a-4105-8acd-b2a04cce8df9-alertmanager-data-science-monitoringstack-db" pod "1294ed53-4e5a-4105-8acd-b2a04cce8df9" (UID: "1294ed53-4e5a-4105-8acd-b2a04cce8df9"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:38:03.797867 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:03.797849 5274 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/1294ed53-4e5a-4105-8acd-b2a04cce8df9-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:38:03.799819 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:03.799789 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1294ed53-4e5a-4105-8acd-b2a04cce8df9-config-volume" pod "1294ed53-4e5a-4105-8acd-b2a04cce8df9" (UID: "1294ed53-4e5a-4105-8acd-b2a04cce8df9"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:38:03.800162 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:03.800120 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1294ed53-4e5a-4105-8acd-b2a04cce8df9-kube-api-access-b7jqx" pod "1294ed53-4e5a-4105-8acd-b2a04cce8df9" (UID: "1294ed53-4e5a-4105-8acd-b2a04cce8df9"). InnerVolumeSpecName "kube-api-access-b7jqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:38:03.800276 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:03.800203 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1294ed53-4e5a-4105-8acd-b2a04cce8df9-tls-assets" pod "1294ed53-4e5a-4105-8acd-b2a04cce8df9" (UID: "1294ed53-4e5a-4105-8acd-b2a04cce8df9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:38:03.800276 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:03.800206 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1294ed53-4e5a-4105-8acd-b2a04cce8df9-config-out" pod "1294ed53-4e5a-4105-8acd-b2a04cce8df9" (UID: "1294ed53-4e5a-4105-8acd-b2a04cce8df9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:38:03.803696 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:03.803672 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1294ed53-4e5a-4105-8acd-b2a04cce8df9-cluster-tls-config" pod "1294ed53-4e5a-4105-8acd-b2a04cce8df9" (UID: "1294ed53-4e5a-4105-8acd-b2a04cce8df9"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:38:03.810790 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:03.810768 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1294ed53-4e5a-4105-8acd-b2a04cce8df9-web-config" pod "1294ed53-4e5a-4105-8acd-b2a04cce8df9" (UID: "1294ed53-4e5a-4105-8acd-b2a04cce8df9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:38:03.898323 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:03.898305 5274 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1294ed53-4e5a-4105-8acd-b2a04cce8df9-tls-assets\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:38:03.898427 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:03.898324 5274 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1294ed53-4e5a-4105-8acd-b2a04cce8df9-config-volume\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:38:03.898427 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:03.898334 5274 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1294ed53-4e5a-4105-8acd-b2a04cce8df9-cluster-tls-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:38:03.898427 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:03.898343 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7jqx\" (UniqueName: \"kubernetes.io/projected/1294ed53-4e5a-4105-8acd-b2a04cce8df9-kube-api-access-b7jqx\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:38:03.898427 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:03.898353 5274 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1294ed53-4e5a-4105-8acd-b2a04cce8df9-web-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:38:03.898427 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:03.898361 5274 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1294ed53-4e5a-4105-8acd-b2a04cce8df9-config-out\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:38:04.333036 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:04.333000 5274 generic.go:356] "Generic (PLEG): container finished" podID="1294ed53-4e5a-4105-8acd-b2a04cce8df9" containerID="8336c41d20990947f1b50c861028d66de9cc789fdb475a52a5d50d2e6033a51b" exitCode=0 Oct 31 00:38:04.333494 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:04.333075 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:38:04.333494 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:04.333085 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"1294ed53-4e5a-4105-8acd-b2a04cce8df9","Type":"ContainerDied","Data":"8336c41d20990947f1b50c861028d66de9cc789fdb475a52a5d50d2e6033a51b"} Oct 31 00:38:04.333494 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:04.333148 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"1294ed53-4e5a-4105-8acd-b2a04cce8df9","Type":"ContainerDied","Data":"132d06a037a86cdaade8a9f9e97cdd4b17d1dfaf9cf697935409ece0e5dcfe01"} Oct 31 00:38:04.333494 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:04.333176 5274 scope.go:122] "RemoveContainer" containerID="74f50f1c5d3b2e1f48497e1aee716fbc62b70202794e48526ecf56578c8a6837" Oct 31 00:38:04.342742 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:04.342726 5274 scope.go:122] "RemoveContainer" containerID="8336c41d20990947f1b50c861028d66de9cc789fdb475a52a5d50d2e6033a51b" Oct 31 00:38:04.350586 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:04.350566 5274 scope.go:122] "RemoveContainer" containerID="d9a54253cea9c7ef6818bc231a5416a8c09dbcd5294c96a097b3f3f3dbda105c" Oct 31 00:38:04.358549 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:04.358529 5274 scope.go:122] "RemoveContainer" containerID="74f50f1c5d3b2e1f48497e1aee716fbc62b70202794e48526ecf56578c8a6837" Oct 31 00:38:04.358755 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:04.358738 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:38:04.358837 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:38:04.358819 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74f50f1c5d3b2e1f48497e1aee716fbc62b70202794e48526ecf56578c8a6837\": container with ID starting with 74f50f1c5d3b2e1f48497e1aee716fbc62b70202794e48526ecf56578c8a6837 not found: ID does not exist" containerID="74f50f1c5d3b2e1f48497e1aee716fbc62b70202794e48526ecf56578c8a6837" Oct 31 00:38:04.358875 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:04.358846 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74f50f1c5d3b2e1f48497e1aee716fbc62b70202794e48526ecf56578c8a6837"} err="failed to get container status \"74f50f1c5d3b2e1f48497e1aee716fbc62b70202794e48526ecf56578c8a6837\": rpc error: code = NotFound desc = could not find container \"74f50f1c5d3b2e1f48497e1aee716fbc62b70202794e48526ecf56578c8a6837\": container with ID starting with 74f50f1c5d3b2e1f48497e1aee716fbc62b70202794e48526ecf56578c8a6837 not found: ID does not exist" Oct 31 00:38:04.358875 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:04.358867 5274 scope.go:122] "RemoveContainer" containerID="8336c41d20990947f1b50c861028d66de9cc789fdb475a52a5d50d2e6033a51b" Oct 31 00:38:04.359189 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:38:04.359171 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8336c41d20990947f1b50c861028d66de9cc789fdb475a52a5d50d2e6033a51b\": container with ID starting with 8336c41d20990947f1b50c861028d66de9cc789fdb475a52a5d50d2e6033a51b not found: ID does not exist" containerID="8336c41d20990947f1b50c861028d66de9cc789fdb475a52a5d50d2e6033a51b" Oct 31 00:38:04.359231 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:04.359197 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8336c41d20990947f1b50c861028d66de9cc789fdb475a52a5d50d2e6033a51b"} err="failed to get container status \"8336c41d20990947f1b50c861028d66de9cc789fdb475a52a5d50d2e6033a51b\": rpc error: code = NotFound desc = could not find container \"8336c41d20990947f1b50c861028d66de9cc789fdb475a52a5d50d2e6033a51b\": container with ID starting with 8336c41d20990947f1b50c861028d66de9cc789fdb475a52a5d50d2e6033a51b not found: ID does not exist" Oct 31 00:38:04.359231 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:04.359211 5274 scope.go:122] "RemoveContainer" containerID="d9a54253cea9c7ef6818bc231a5416a8c09dbcd5294c96a097b3f3f3dbda105c" Oct 31 00:38:04.359430 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:38:04.359409 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9a54253cea9c7ef6818bc231a5416a8c09dbcd5294c96a097b3f3f3dbda105c\": container with ID starting with d9a54253cea9c7ef6818bc231a5416a8c09dbcd5294c96a097b3f3f3dbda105c not found: ID does not exist" containerID="d9a54253cea9c7ef6818bc231a5416a8c09dbcd5294c96a097b3f3f3dbda105c" Oct 31 00:38:04.359524 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:04.359433 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9a54253cea9c7ef6818bc231a5416a8c09dbcd5294c96a097b3f3f3dbda105c"} err="failed to get container status \"d9a54253cea9c7ef6818bc231a5416a8c09dbcd5294c96a097b3f3f3dbda105c\": rpc error: code = NotFound desc = could not find container \"d9a54253cea9c7ef6818bc231a5416a8c09dbcd5294c96a097b3f3f3dbda105c\": container with ID starting with d9a54253cea9c7ef6818bc231a5416a8c09dbcd5294c96a097b3f3f3dbda105c not found: ID does not exist" Oct 31 00:38:04.363629 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:04.363607 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:38:05.480767 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:05.480733 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1294ed53-4e5a-4105-8acd-b2a04cce8df9" path="/var/lib/kubelet/pods/1294ed53-4e5a-4105-8acd-b2a04cce8df9/volumes" Oct 31 00:38:07.380539 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.380504 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:38:07.381103 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.381084 5274 cpu_manager.go:414] "containerMap: removing container" podUID="1294ed53-4e5a-4105-8acd-b2a04cce8df9" containerName="init-config-reloader" Oct 31 00:38:07.381189 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.381104 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1294ed53-4e5a-4105-8acd-b2a04cce8df9" containerName="init-config-reloader" Oct 31 00:38:07.381189 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.381119 5274 cpu_manager.go:414] "containerMap: removing container" podUID="1294ed53-4e5a-4105-8acd-b2a04cce8df9" containerName="alertmanager" Oct 31 00:38:07.381189 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.381127 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1294ed53-4e5a-4105-8acd-b2a04cce8df9" containerName="alertmanager" Oct 31 00:38:07.381189 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.381139 5274 cpu_manager.go:414] "containerMap: removing container" podUID="1294ed53-4e5a-4105-8acd-b2a04cce8df9" containerName="config-reloader" Oct 31 00:38:07.381189 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.381146 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1294ed53-4e5a-4105-8acd-b2a04cce8df9" containerName="config-reloader" Oct 31 00:38:07.381460 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.381267 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="1294ed53-4e5a-4105-8acd-b2a04cce8df9" containerName="alertmanager" Oct 31 00:38:07.381460 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.381279 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="1294ed53-4e5a-4105-8acd-b2a04cce8df9" containerName="config-reloader" Oct 31 00:38:07.386744 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.386723 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:38:07.390749 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.390725 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:38:07.390903 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.390760 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 31 00:38:07.390903 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.390768 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 31 00:38:07.390903 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.390783 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 31 00:38:07.391447 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.391430 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-94cpx\"" Oct 31 00:38:07.397083 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.397062 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:38:07.524807 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.524774 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c815941f-4f86-418a-8d60-e508abc538eb-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c815941f-4f86-418a-8d60-e508abc538eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:38:07.525004 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.524825 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/c815941f-4f86-418a-8d60-e508abc538eb-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c815941f-4f86-418a-8d60-e508abc538eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:38:07.525004 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.524938 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c815941f-4f86-418a-8d60-e508abc538eb-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c815941f-4f86-418a-8d60-e508abc538eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:38:07.525004 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.524984 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c815941f-4f86-418a-8d60-e508abc538eb-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c815941f-4f86-418a-8d60-e508abc538eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:38:07.525120 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.525069 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c815941f-4f86-418a-8d60-e508abc538eb-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c815941f-4f86-418a-8d60-e508abc538eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:38:07.525120 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.525104 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c815941f-4f86-418a-8d60-e508abc538eb-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c815941f-4f86-418a-8d60-e508abc538eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:38:07.525187 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.525166 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqrlz\" (UniqueName: \"kubernetes.io/projected/c815941f-4f86-418a-8d60-e508abc538eb-kube-api-access-jqrlz\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c815941f-4f86-418a-8d60-e508abc538eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:38:07.625814 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.625772 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c815941f-4f86-418a-8d60-e508abc538eb-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c815941f-4f86-418a-8d60-e508abc538eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:38:07.625814 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.625812 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c815941f-4f86-418a-8d60-e508abc538eb-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c815941f-4f86-418a-8d60-e508abc538eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:38:07.626089 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.625830 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c815941f-4f86-418a-8d60-e508abc538eb-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c815941f-4f86-418a-8d60-e508abc538eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:38:07.626089 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.625853 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c815941f-4f86-418a-8d60-e508abc538eb-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c815941f-4f86-418a-8d60-e508abc538eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:38:07.626089 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.625894 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqrlz\" (UniqueName: \"kubernetes.io/projected/c815941f-4f86-418a-8d60-e508abc538eb-kube-api-access-jqrlz\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c815941f-4f86-418a-8d60-e508abc538eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:38:07.626265 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.626080 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c815941f-4f86-418a-8d60-e508abc538eb-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c815941f-4f86-418a-8d60-e508abc538eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:38:07.626265 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.626187 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/c815941f-4f86-418a-8d60-e508abc538eb-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c815941f-4f86-418a-8d60-e508abc538eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:38:07.626550 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.626529 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/c815941f-4f86-418a-8d60-e508abc538eb-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c815941f-4f86-418a-8d60-e508abc538eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:38:07.628479 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.628455 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c815941f-4f86-418a-8d60-e508abc538eb-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c815941f-4f86-418a-8d60-e508abc538eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:38:07.628863 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.628843 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c815941f-4f86-418a-8d60-e508abc538eb-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c815941f-4f86-418a-8d60-e508abc538eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:38:07.629099 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.629080 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c815941f-4f86-418a-8d60-e508abc538eb-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c815941f-4f86-418a-8d60-e508abc538eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:38:07.629169 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.629125 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c815941f-4f86-418a-8d60-e508abc538eb-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c815941f-4f86-418a-8d60-e508abc538eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:38:07.629169 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.629125 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c815941f-4f86-418a-8d60-e508abc538eb-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c815941f-4f86-418a-8d60-e508abc538eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:38:07.638276 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.638207 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqrlz\" (UniqueName: \"kubernetes.io/projected/c815941f-4f86-418a-8d60-e508abc538eb-kube-api-access-jqrlz\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"c815941f-4f86-418a-8d60-e508abc538eb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:38:07.706236 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.706197 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:38:07.824111 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.824075 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:38:07.829313 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.829296 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:07.834385 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.834342 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 31 00:38:07.834927 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.834851 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 31 00:38:07.836524 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.836502 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 31 00:38:07.836630 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.836587 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 31 00:38:07.836734 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.836650 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 31 00:38:07.846935 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.846895 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:38:07.928130 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.928045 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3813ab2d-eb02-48d7-85fe-6c85d327f09e-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:07.928130 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.928093 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3813ab2d-eb02-48d7-85fe-6c85d327f09e-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:07.928309 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.928168 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3813ab2d-eb02-48d7-85fe-6c85d327f09e-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:07.928309 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.928221 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs2n4\" (UniqueName: \"kubernetes.io/projected/3813ab2d-eb02-48d7-85fe-6c85d327f09e-kube-api-access-vs2n4\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:07.928309 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.928246 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3813ab2d-eb02-48d7-85fe-6c85d327f09e-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:07.928426 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.928313 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3813ab2d-eb02-48d7-85fe-6c85d327f09e-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:07.928426 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.928368 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:07.928426 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.928391 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:07.928589 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.928455 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:07.928589 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.928521 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:07.928589 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.928555 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:07.928712 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.928585 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:07.934535 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.934513 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:38:07.934666 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.934614 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 31 00:38:07.934666 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.934657 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:38:07.934761 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.934674 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-r54vf\"" Oct 31 00:38:07.991362 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:07.991337 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:38:07.993150 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:38:07.993118 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc815941f_4f86_418a_8d60_e508abc538eb.slice/crio-ccc721cccaace846272f48426d43ef2e0cdaa83df89236974e6ba3c76a91b75a WatchSource:0}: Error finding container ccc721cccaace846272f48426d43ef2e0cdaa83df89236974e6ba3c76a91b75a: Status 404 returned error can't find the container with id ccc721cccaace846272f48426d43ef2e0cdaa83df89236974e6ba3c76a91b75a Oct 31 00:38:08.029945 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:08.029920 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3813ab2d-eb02-48d7-85fe-6c85d327f09e-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:08.030109 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:08.029981 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3813ab2d-eb02-48d7-85fe-6c85d327f09e-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:08.030109 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:08.030023 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3813ab2d-eb02-48d7-85fe-6c85d327f09e-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:08.030109 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:08.030085 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs2n4\" (UniqueName: \"kubernetes.io/projected/3813ab2d-eb02-48d7-85fe-6c85d327f09e-kube-api-access-vs2n4\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:08.030257 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:08.030113 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3813ab2d-eb02-48d7-85fe-6c85d327f09e-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:08.030257 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:08.030140 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3813ab2d-eb02-48d7-85fe-6c85d327f09e-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:08.030656 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:08.030630 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3813ab2d-eb02-48d7-85fe-6c85d327f09e-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:08.030840 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:08.030812 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3813ab2d-eb02-48d7-85fe-6c85d327f09e-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:08.030975 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:08.030887 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:08.030975 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:08.030916 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3813ab2d-eb02-48d7-85fe-6c85d327f09e-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:08.031094 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:08.030997 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:08.031094 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:08.031048 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:08.031094 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:08.031088 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:08.031277 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:08.031115 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:08.031277 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:08.031142 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:08.032757 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:08.032706 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3813ab2d-eb02-48d7-85fe-6c85d327f09e-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:08.033416 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:08.033393 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3813ab2d-eb02-48d7-85fe-6c85d327f09e-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:08.034045 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:08.034001 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:08.034144 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:08.034104 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:08.034312 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:08.034288 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:08.034371 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:08.034322 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:08.035107 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:08.035091 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:08.035547 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:08.035531 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:08.041720 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:08.041701 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs2n4\" (UniqueName: \"kubernetes.io/projected/3813ab2d-eb02-48d7-85fe-6c85d327f09e-kube-api-access-vs2n4\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:08.363925 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:08.363889 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"c815941f-4f86-418a-8d60-e508abc538eb","Type":"ContainerStarted","Data":"47f94b4ba825bd6c27c3475870294df290c0d6be7e4a9cce2935207be4a74368"} Oct 31 00:38:08.364088 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:08.363971 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"c815941f-4f86-418a-8d60-e508abc538eb","Type":"ContainerStarted","Data":"ccc721cccaace846272f48426d43ef2e0cdaa83df89236974e6ba3c76a91b75a"} Oct 31 00:38:09.336161 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:09.336132 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jrq79" Oct 31 00:38:10.856129 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:10.856095 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:10.864614 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:10.864583 5274 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:10.956919 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:10.956893 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:10.957057 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:10.957010 5274 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:10.960089 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:10.960071 5274 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") DevicePath \"csi-51e8027f163b5f98040faf3f8c30848dd44a7a90ca2bd0af2446ea6398ed018d\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:11.050153 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:11.050130 5274 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/53bcf0e8434a816dcc013cea20657add42b4da36ecdc0fff46af7413bc0037a5/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:11.064925 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:11.064896 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:11.153671 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:11.153643 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:11.439533 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:11.439511 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:38:11.441300 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:38:11.441271 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3813ab2d_eb02_48d7_85fe_6c85d327f09e.slice/crio-1d950e7297e8daee88b1f281acedb0279085326129e65a980a42fa64a16763e3 WatchSource:0}: Error finding container 1d950e7297e8daee88b1f281acedb0279085326129e65a980a42fa64a16763e3: Status 404 returned error can't find the container with id 1d950e7297e8daee88b1f281acedb0279085326129e65a980a42fa64a16763e3 Oct 31 00:38:12.388090 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:12.388054 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"3813ab2d-eb02-48d7-85fe-6c85d327f09e","Type":"ContainerStarted","Data":"eca7b05bc70871dd2d81521c384c0ab583ab92056a90b67fb595bde33307111a"} Oct 31 00:38:12.388487 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:12.388138 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"3813ab2d-eb02-48d7-85fe-6c85d327f09e","Type":"ContainerStarted","Data":"1d950e7297e8daee88b1f281acedb0279085326129e65a980a42fa64a16763e3"} Oct 31 00:38:12.389659 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:12.389634 5274 generic.go:356] "Generic (PLEG): container finished" podID="c815941f-4f86-418a-8d60-e508abc538eb" containerID="47f94b4ba825bd6c27c3475870294df290c0d6be7e4a9cce2935207be4a74368" exitCode=0 Oct 31 00:38:12.389772 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:12.389702 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"c815941f-4f86-418a-8d60-e508abc538eb","Type":"ContainerDied","Data":"47f94b4ba825bd6c27c3475870294df290c0d6be7e4a9cce2935207be4a74368"} Oct 31 00:38:13.399445 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:13.399409 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"c815941f-4f86-418a-8d60-e508abc538eb","Type":"ContainerStarted","Data":"e6f9b9d9a577cc0fcd9018f89f56d6092b781a68cc1c1b12ed9e63ac448d9df1"} Oct 31 00:38:13.399858 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:13.399474 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"c815941f-4f86-418a-8d60-e508abc538eb","Type":"ContainerStarted","Data":"7087fa10a92db40382436bc890574981812d74c122b4e3325b89b8d6a17026ce"} Oct 31 00:38:13.399858 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:13.399819 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:38:13.426877 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:13.426827 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=3.240912114 podStartE2EDuration="6.426810179s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.185898065s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:13.425158362 +0000 UTC m=+2330.061977079" watchObservedRunningTime="2026-10-31 00:38:13.426810179 +0000 UTC m=+2330.063628897" Oct 31 00:38:14.335133 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:14.335094 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-9c4wg" Oct 31 00:38:15.411668 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:15.411638 5274 generic.go:356] "Generic (PLEG): container finished" podID="3813ab2d-eb02-48d7-85fe-6c85d327f09e" containerID="eca7b05bc70871dd2d81521c384c0ab583ab92056a90b67fb595bde33307111a" exitCode=0 Oct 31 00:38:15.412010 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:15.411708 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"3813ab2d-eb02-48d7-85fe-6c85d327f09e","Type":"ContainerDied","Data":"eca7b05bc70871dd2d81521c384c0ab583ab92056a90b67fb595bde33307111a"} Oct 31 00:38:16.419444 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:16.419413 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"3813ab2d-eb02-48d7-85fe-6c85d327f09e","Type":"ContainerStarted","Data":"ef15a038a53758d8a9b451b846c4b280ca74720295209ee52b54205e5cce9e69"} Oct 31 00:38:16.419805 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:16.419476 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"3813ab2d-eb02-48d7-85fe-6c85d327f09e","Type":"ContainerStarted","Data":"662ba948f1acf0026c92a94a2e58228349dd237df6185d12e4e89b851d766cd9"} Oct 31 00:38:16.419805 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:16.419500 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"3813ab2d-eb02-48d7-85fe-6c85d327f09e","Type":"ContainerStarted","Data":"f7fbe6f697b6b51dd7028cc27c922b46b0762d5953332c396d05cf4aa8f25c8d"} Oct 31 00:38:16.452774 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:16.452725 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=6.066038786 podStartE2EDuration="9.452711794s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.386673008s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:16.45116628 +0000 UTC m=+2333.087984997" watchObservedRunningTime="2026-10-31 00:38:16.452711794 +0000 UTC m=+2333.089530510" Oct 31 00:38:19.406682 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:19.406653 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:38:20.898641 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:38:20.898605 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:38:21.154173 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:21.154097 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:26.154023 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:26.153992 5274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:26.159182 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:26.159159 5274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:26.474290 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:38:26.474224 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:38:50.899271 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:38:50.899234 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:39:20.900069 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:39:20.900030 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:39:32.049446 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.049411 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jrq79"] Oct 31 00:39:32.049927 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.049625 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jrq79" podUID="37c7d1c2-01c8-47f3-8e6c-7b14918b6743" containerName="kube-rbac-proxy" containerID="cri-o://4fd48a1469c39f8446954a1fb2d2408b628676ff83dcbfce418b835c6967278f" gracePeriod=30 Oct 31 00:39:32.069365 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.069341 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-9c4wg"] Oct 31 00:39:32.069529 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.069510 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-9c4wg" podUID="1db65176-7e26-48d5-b66d-ea279099c362" containerName="korrel8r" containerID="cri-o://9f420f1318939dfee379bbb9881ed3978aa5d3410e7ceab2f19026fc21b0959a" gracePeriod=30 Oct 31 00:39:32.265587 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.265547 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:39:32.266106 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.266051 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="3813ab2d-eb02-48d7-85fe-6c85d327f09e" containerName="prometheus" containerID="cri-o://f7fbe6f697b6b51dd7028cc27c922b46b0762d5953332c396d05cf4aa8f25c8d" gracePeriod=600 Oct 31 00:39:32.267429 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.266531 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="3813ab2d-eb02-48d7-85fe-6c85d327f09e" containerName="thanos-sidecar" containerID="cri-o://ef15a038a53758d8a9b451b846c4b280ca74720295209ee52b54205e5cce9e69" gracePeriod=600 Oct 31 00:39:32.267705 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.266567 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="3813ab2d-eb02-48d7-85fe-6c85d327f09e" containerName="config-reloader" containerID="cri-o://662ba948f1acf0026c92a94a2e58228349dd237df6185d12e4e89b851d766cd9" gracePeriod=600 Oct 31 00:39:32.276031 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.275201 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:39:32.276031 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.275387 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="c815941f-4f86-418a-8d60-e508abc538eb" containerName="alertmanager" containerID="cri-o://7087fa10a92db40382436bc890574981812d74c122b4e3325b89b8d6a17026ce" gracePeriod=120 Oct 31 00:39:32.277378 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.276638 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="c815941f-4f86-418a-8d60-e508abc538eb" containerName="config-reloader" containerID="cri-o://e6f9b9d9a577cc0fcd9018f89f56d6092b781a68cc1c1b12ed9e63ac448d9df1" gracePeriod=120 Oct 31 00:39:32.435909 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.435875 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:39:32.436212 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.436104 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="5ed80a10-b519-4d87-a4a4-0783c84af9aa" containerName="otc-container" containerID="cri-o://c5089e368c3cd83016b26e69d547360b47bfc6a4fe4fb0048916e6c9a9bdd325" gracePeriod=30 Oct 31 00:39:32.584769 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.584703 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-9c4wg" Oct 31 00:39:32.601388 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.599426 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jrq79" Oct 31 00:39:32.667302 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.667270 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:39:32.738576 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.738546 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37c7d1c2-01c8-47f3-8e6c-7b14918b6743-kube-api-access-lhf5v\" (UniqueName: \"kubernetes.io/projected/37c7d1c2-01c8-47f3-8e6c-7b14918b6743-kube-api-access-lhf5v\") pod \"37c7d1c2-01c8-47f3-8e6c-7b14918b6743\" (UID: \"37c7d1c2-01c8-47f3-8e6c-7b14918b6743\") " Oct 31 00:39:32.738738 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.738610 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37c7d1c2-01c8-47f3-8e6c-7b14918b6743-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/37c7d1c2-01c8-47f3-8e6c-7b14918b6743-prometheus-client-cert\") pod \"37c7d1c2-01c8-47f3-8e6c-7b14918b6743\" (UID: \"37c7d1c2-01c8-47f3-8e6c-7b14918b6743\") " Oct 31 00:39:32.738738 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.738663 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37c7d1c2-01c8-47f3-8e6c-7b14918b6743-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/37c7d1c2-01c8-47f3-8e6c-7b14918b6743-prometheus-ca\") pod \"37c7d1c2-01c8-47f3-8e6c-7b14918b6743\" (UID: \"37c7d1c2-01c8-47f3-8e6c-7b14918b6743\") " Oct 31 00:39:32.738738 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.738700 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1db65176-7e26-48d5-b66d-ea279099c362-tls\" (UniqueName: \"kubernetes.io/secret/1db65176-7e26-48d5-b66d-ea279099c362-tls\") pod \"1db65176-7e26-48d5-b66d-ea279099c362\" (UID: \"1db65176-7e26-48d5-b66d-ea279099c362\") " Oct 31 00:39:32.738914 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.738741 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1db65176-7e26-48d5-b66d-ea279099c362-kube-api-access-hkhzl\" (UniqueName: \"kubernetes.io/projected/1db65176-7e26-48d5-b66d-ea279099c362-kube-api-access-hkhzl\") pod \"1db65176-7e26-48d5-b66d-ea279099c362\" (UID: \"1db65176-7e26-48d5-b66d-ea279099c362\") " Oct 31 00:39:32.738914 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.738782 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37c7d1c2-01c8-47f3-8e6c-7b14918b6743-tls\" (UniqueName: \"kubernetes.io/secret/37c7d1c2-01c8-47f3-8e6c-7b14918b6743-tls\") pod \"37c7d1c2-01c8-47f3-8e6c-7b14918b6743\" (UID: \"37c7d1c2-01c8-47f3-8e6c-7b14918b6743\") " Oct 31 00:39:32.738914 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.738884 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1db65176-7e26-48d5-b66d-ea279099c362-config\" (UniqueName: \"kubernetes.io/configmap/1db65176-7e26-48d5-b66d-ea279099c362-config\") pod \"1db65176-7e26-48d5-b66d-ea279099c362\" (UID: \"1db65176-7e26-48d5-b66d-ea279099c362\") " Oct 31 00:39:32.739138 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.738915 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37c7d1c2-01c8-47f3-8e6c-7b14918b6743-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37c7d1c2-01c8-47f3-8e6c-7b14918b6743-kube-rbac-proxy-config\") pod \"37c7d1c2-01c8-47f3-8e6c-7b14918b6743\" (UID: \"37c7d1c2-01c8-47f3-8e6c-7b14918b6743\") " Oct 31 00:39:32.739195 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.739151 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:39:32.739542 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.739511 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37c7d1c2-01c8-47f3-8e6c-7b14918b6743-prometheus-ca" pod "37c7d1c2-01c8-47f3-8e6c-7b14918b6743" (UID: "37c7d1c2-01c8-47f3-8e6c-7b14918b6743"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:39:32.740053 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.740025 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1db65176-7e26-48d5-b66d-ea279099c362-config" pod "1db65176-7e26-48d5-b66d-ea279099c362" (UID: "1db65176-7e26-48d5-b66d-ea279099c362"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:39:32.744464 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.744439 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37c7d1c2-01c8-47f3-8e6c-7b14918b6743-prometheus-client-cert" pod "37c7d1c2-01c8-47f3-8e6c-7b14918b6743" (UID: "37c7d1c2-01c8-47f3-8e6c-7b14918b6743"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:32.744566 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.744477 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37c7d1c2-01c8-47f3-8e6c-7b14918b6743-kube-api-access-lhf5v" pod "37c7d1c2-01c8-47f3-8e6c-7b14918b6743" (UID: "37c7d1c2-01c8-47f3-8e6c-7b14918b6743"). InnerVolumeSpecName "kube-api-access-lhf5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:39:32.744566 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.744491 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37c7d1c2-01c8-47f3-8e6c-7b14918b6743-tls" pod "37c7d1c2-01c8-47f3-8e6c-7b14918b6743" (UID: "37c7d1c2-01c8-47f3-8e6c-7b14918b6743"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:32.744813 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.744791 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1db65176-7e26-48d5-b66d-ea279099c362-kube-api-access-hkhzl" pod "1db65176-7e26-48d5-b66d-ea279099c362" (UID: "1db65176-7e26-48d5-b66d-ea279099c362"). InnerVolumeSpecName "kube-api-access-hkhzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:39:32.744909 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.744850 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37c7d1c2-01c8-47f3-8e6c-7b14918b6743-kube-rbac-proxy-config" pod "37c7d1c2-01c8-47f3-8e6c-7b14918b6743" (UID: "37c7d1c2-01c8-47f3-8e6c-7b14918b6743"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:32.744909 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.744885 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1db65176-7e26-48d5-b66d-ea279099c362-tls" pod "1db65176-7e26-48d5-b66d-ea279099c362" (UID: "1db65176-7e26-48d5-b66d-ea279099c362"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:32.800378 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.800345 5274 generic.go:356] "Generic (PLEG): container finished" podID="1db65176-7e26-48d5-b66d-ea279099c362" containerID="9f420f1318939dfee379bbb9881ed3978aa5d3410e7ceab2f19026fc21b0959a" exitCode=0 Oct 31 00:39:32.800567 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.800392 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-9c4wg" Oct 31 00:39:32.800567 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.800427 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-9c4wg" event={"ID":"1db65176-7e26-48d5-b66d-ea279099c362","Type":"ContainerDied","Data":"9f420f1318939dfee379bbb9881ed3978aa5d3410e7ceab2f19026fc21b0959a"} Oct 31 00:39:32.800567 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.800488 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-9c4wg" event={"ID":"1db65176-7e26-48d5-b66d-ea279099c362","Type":"ContainerDied","Data":"d3cb65881ee763a86df1ee51e962d614c740f9d799048d9df1636cf7e6a08a36"} Oct 31 00:39:32.800567 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.800517 5274 scope.go:122] "RemoveContainer" containerID="9f420f1318939dfee379bbb9881ed3978aa5d3410e7ceab2f19026fc21b0959a" Oct 31 00:39:32.802128 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.802020 5274 generic.go:356] "Generic (PLEG): container finished" podID="5ed80a10-b519-4d87-a4a4-0783c84af9aa" containerID="c5089e368c3cd83016b26e69d547360b47bfc6a4fe4fb0048916e6c9a9bdd325" exitCode=0 Oct 31 00:39:32.802128 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.802063 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:39:32.802128 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.802114 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5ed80a10-b519-4d87-a4a4-0783c84af9aa","Type":"ContainerDied","Data":"c5089e368c3cd83016b26e69d547360b47bfc6a4fe4fb0048916e6c9a9bdd325"} Oct 31 00:39:32.802363 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.802157 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5ed80a10-b519-4d87-a4a4-0783c84af9aa","Type":"ContainerDied","Data":"bcfad1db1d5ecd5bec24310dad6d9074c83f16c64000dfc4c7b67bf940b67731"} Oct 31 00:39:32.804803 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.804764 5274 generic.go:356] "Generic (PLEG): container finished" podID="3813ab2d-eb02-48d7-85fe-6c85d327f09e" containerID="ef15a038a53758d8a9b451b846c4b280ca74720295209ee52b54205e5cce9e69" exitCode=0 Oct 31 00:39:32.804803 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.804782 5274 generic.go:356] "Generic (PLEG): container finished" podID="3813ab2d-eb02-48d7-85fe-6c85d327f09e" containerID="662ba948f1acf0026c92a94a2e58228349dd237df6185d12e4e89b851d766cd9" exitCode=0 Oct 31 00:39:32.804803 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.804793 5274 generic.go:356] "Generic (PLEG): container finished" podID="3813ab2d-eb02-48d7-85fe-6c85d327f09e" containerID="f7fbe6f697b6b51dd7028cc27c922b46b0762d5953332c396d05cf4aa8f25c8d" exitCode=0 Oct 31 00:39:32.805053 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.804854 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 31 00:39:32.805053 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.804790 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"3813ab2d-eb02-48d7-85fe-6c85d327f09e","Type":"ContainerDied","Data":"ef15a038a53758d8a9b451b846c4b280ca74720295209ee52b54205e5cce9e69"} Oct 31 00:39:32.805053 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.805034 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"3813ab2d-eb02-48d7-85fe-6c85d327f09e","Type":"ContainerDied","Data":"662ba948f1acf0026c92a94a2e58228349dd237df6185d12e4e89b851d766cd9"} Oct 31 00:39:32.805204 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.805076 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"3813ab2d-eb02-48d7-85fe-6c85d327f09e","Type":"ContainerDied","Data":"f7fbe6f697b6b51dd7028cc27c922b46b0762d5953332c396d05cf4aa8f25c8d"} Oct 31 00:39:32.805204 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.805115 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"3813ab2d-eb02-48d7-85fe-6c85d327f09e","Type":"ContainerDied","Data":"1d950e7297e8daee88b1f281acedb0279085326129e65a980a42fa64a16763e3"} Oct 31 00:39:32.808285 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.808263 5274 generic.go:356] "Generic (PLEG): container finished" podID="c815941f-4f86-418a-8d60-e508abc538eb" containerID="e6f9b9d9a577cc0fcd9018f89f56d6092b781a68cc1c1b12ed9e63ac448d9df1" exitCode=0 Oct 31 00:39:32.808397 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.808341 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"c815941f-4f86-418a-8d60-e508abc538eb","Type":"ContainerDied","Data":"e6f9b9d9a577cc0fcd9018f89f56d6092b781a68cc1c1b12ed9e63ac448d9df1"} Oct 31 00:39:32.809786 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.809761 5274 generic.go:356] "Generic (PLEG): container finished" podID="37c7d1c2-01c8-47f3-8e6c-7b14918b6743" containerID="4fd48a1469c39f8446954a1fb2d2408b628676ff83dcbfce418b835c6967278f" exitCode=0 Oct 31 00:39:32.809898 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.809804 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jrq79" Oct 31 00:39:32.809898 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.809803 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jrq79" event={"ID":"37c7d1c2-01c8-47f3-8e6c-7b14918b6743","Type":"ContainerDied","Data":"4fd48a1469c39f8446954a1fb2d2408b628676ff83dcbfce418b835c6967278f"} Oct 31 00:39:32.809898 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.809852 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jrq79" event={"ID":"37c7d1c2-01c8-47f3-8e6c-7b14918b6743","Type":"ContainerDied","Data":"d18e4dbfb10cbd5e757d8b87d15603abfd816f08f7b9d1657ced2b30562ead2e"} Oct 31 00:39:32.811494 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.811475 5274 scope.go:122] "RemoveContainer" containerID="9f420f1318939dfee379bbb9881ed3978aa5d3410e7ceab2f19026fc21b0959a" Oct 31 00:39:32.811757 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:39:32.811739 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f420f1318939dfee379bbb9881ed3978aa5d3410e7ceab2f19026fc21b0959a\": container with ID starting with 9f420f1318939dfee379bbb9881ed3978aa5d3410e7ceab2f19026fc21b0959a not found: ID does not exist" containerID="9f420f1318939dfee379bbb9881ed3978aa5d3410e7ceab2f19026fc21b0959a" Oct 31 00:39:32.811827 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.811763 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f420f1318939dfee379bbb9881ed3978aa5d3410e7ceab2f19026fc21b0959a"} err="failed to get container status \"9f420f1318939dfee379bbb9881ed3978aa5d3410e7ceab2f19026fc21b0959a\": rpc error: code = NotFound desc = could not find container \"9f420f1318939dfee379bbb9881ed3978aa5d3410e7ceab2f19026fc21b0959a\": container with ID starting with 9f420f1318939dfee379bbb9881ed3978aa5d3410e7ceab2f19026fc21b0959a not found: ID does not exist" Oct 31 00:39:32.811827 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.811779 5274 scope.go:122] "RemoveContainer" containerID="c5089e368c3cd83016b26e69d547360b47bfc6a4fe4fb0048916e6c9a9bdd325" Oct 31 00:39:32.821145 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.821129 5274 scope.go:122] "RemoveContainer" containerID="c5089e368c3cd83016b26e69d547360b47bfc6a4fe4fb0048916e6c9a9bdd325" Oct 31 00:39:32.821391 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:39:32.821372 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5089e368c3cd83016b26e69d547360b47bfc6a4fe4fb0048916e6c9a9bdd325\": container with ID starting with c5089e368c3cd83016b26e69d547360b47bfc6a4fe4fb0048916e6c9a9bdd325 not found: ID does not exist" containerID="c5089e368c3cd83016b26e69d547360b47bfc6a4fe4fb0048916e6c9a9bdd325" Oct 31 00:39:32.821452 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.821397 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5089e368c3cd83016b26e69d547360b47bfc6a4fe4fb0048916e6c9a9bdd325"} err="failed to get container status \"c5089e368c3cd83016b26e69d547360b47bfc6a4fe4fb0048916e6c9a9bdd325\": rpc error: code = NotFound desc = could not find container \"c5089e368c3cd83016b26e69d547360b47bfc6a4fe4fb0048916e6c9a9bdd325\": container with ID starting with c5089e368c3cd83016b26e69d547360b47bfc6a4fe4fb0048916e6c9a9bdd325 not found: ID does not exist" Oct 31 00:39:32.821452 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.821420 5274 scope.go:122] "RemoveContainer" containerID="ef15a038a53758d8a9b451b846c4b280ca74720295209ee52b54205e5cce9e69" Oct 31 00:39:32.828996 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.828980 5274 scope.go:122] "RemoveContainer" containerID="662ba948f1acf0026c92a94a2e58228349dd237df6185d12e4e89b851d766cd9" Oct 31 00:39:32.835047 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.834974 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-9c4wg"] Oct 31 00:39:32.838686 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.838668 5274 scope.go:122] "RemoveContainer" containerID="f7fbe6f697b6b51dd7028cc27c922b46b0762d5953332c396d05cf4aa8f25c8d" Oct 31 00:39:32.839557 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.839495 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ed80a10-b519-4d87-a4a4-0783c84af9aa-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ed80a10-b519-4d87-a4a4-0783c84af9aa-monitoring-tls-certs\") pod \"5ed80a10-b519-4d87-a4a4-0783c84af9aa\" (UID: \"5ed80a10-b519-4d87-a4a4-0783c84af9aa\") " Oct 31 00:39:32.839557 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.839542 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ed80a10-b519-4d87-a4a4-0783c84af9aa-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ed80a10-b519-4d87-a4a4-0783c84af9aa-otlp-tls-certs\") pod \"5ed80a10-b519-4d87-a4a4-0783c84af9aa\" (UID: \"5ed80a10-b519-4d87-a4a4-0783c84af9aa\") " Oct 31 00:39:32.839809 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.839575 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ed80a10-b519-4d87-a4a4-0783c84af9aa-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5ed80a10-b519-4d87-a4a4-0783c84af9aa-data-science-collector-ta-client-cert\") pod \"5ed80a10-b519-4d87-a4a4-0783c84af9aa\" (UID: \"5ed80a10-b519-4d87-a4a4-0783c84af9aa\") " Oct 31 00:39:32.839809 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.839621 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3813ab2d-eb02-48d7-85fe-6c85d327f09e-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3813ab2d-eb02-48d7-85fe-6c85d327f09e-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " Oct 31 00:39:32.839809 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.839652 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ed80a10-b519-4d87-a4a4-0783c84af9aa-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ed80a10-b519-4d87-a4a4-0783c84af9aa-tls-certs\") pod \"5ed80a10-b519-4d87-a4a4-0783c84af9aa\" (UID: \"5ed80a10-b519-4d87-a4a4-0783c84af9aa\") " Oct 31 00:39:32.839809 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.839695 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-secret-prometheus-web-tls-ca\") pod \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " Oct 31 00:39:32.839809 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.839722 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-web-config\" (UniqueName: \"kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-web-config\") pod \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " Oct 31 00:39:32.839809 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.839752 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3813ab2d-eb02-48d7-85fe-6c85d327f09e-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3813ab2d-eb02-48d7-85fe-6c85d327f09e-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " Oct 31 00:39:32.839809 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.839792 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3813ab2d-eb02-48d7-85fe-6c85d327f09e-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3813ab2d-eb02-48d7-85fe-6c85d327f09e-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " Oct 31 00:39:32.840381 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.839821 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3813ab2d-eb02-48d7-85fe-6c85d327f09e-config-out\" (UniqueName: \"kubernetes.io/empty-dir/3813ab2d-eb02-48d7-85fe-6c85d327f09e-config-out\") pod \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " Oct 31 00:39:32.840381 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.839848 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3813ab2d-eb02-48d7-85fe-6c85d327f09e-tls-assets\" (UniqueName: \"kubernetes.io/projected/3813ab2d-eb02-48d7-85fe-6c85d327f09e-tls-assets\") pod \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " Oct 31 00:39:32.840381 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.839883 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " Oct 31 00:39:32.840381 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.839927 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ed80a10-b519-4d87-a4a4-0783c84af9aa-kube-api-access-8cfsd\" (UniqueName: \"kubernetes.io/projected/5ed80a10-b519-4d87-a4a4-0783c84af9aa-kube-api-access-8cfsd\") pod \"5ed80a10-b519-4d87-a4a4-0783c84af9aa\" (UID: \"5ed80a10-b519-4d87-a4a4-0783c84af9aa\") " Oct 31 00:39:32.840381 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.839981 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3813ab2d-eb02-48d7-85fe-6c85d327f09e-kube-api-access-vs2n4\" (UniqueName: \"kubernetes.io/projected/3813ab2d-eb02-48d7-85fe-6c85d327f09e-kube-api-access-vs2n4\") pod \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " Oct 31 00:39:32.840381 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.840008 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-config\" (UniqueName: \"kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-config\") pod \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " Oct 31 00:39:32.840381 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.840059 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-thanos-prometheus-http-client-file\") pod \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " Oct 31 00:39:32.840381 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.840146 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") pod \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " Oct 31 00:39:32.840381 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.840185 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ed80a10-b519-4d87-a4a4-0783c84af9aa-otc-internal\" (UniqueName: \"kubernetes.io/configmap/5ed80a10-b519-4d87-a4a4-0783c84af9aa-otc-internal\") pod \"5ed80a10-b519-4d87-a4a4-0783c84af9aa\" (UID: \"5ed80a10-b519-4d87-a4a4-0783c84af9aa\") " Oct 31 00:39:32.840381 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.840235 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\" (UID: \"3813ab2d-eb02-48d7-85fe-6c85d327f09e\") " Oct 31 00:39:32.841210 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.840533 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1db65176-7e26-48d5-b66d-ea279099c362-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.841210 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.840549 5274 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37c7d1c2-01c8-47f3-8e6c-7b14918b6743-kube-rbac-proxy-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.841210 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.840562 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhf5v\" (UniqueName: \"kubernetes.io/projected/37c7d1c2-01c8-47f3-8e6c-7b14918b6743-kube-api-access-lhf5v\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.841210 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.840577 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/37c7d1c2-01c8-47f3-8e6c-7b14918b6743-prometheus-client-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.841210 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.840593 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/37c7d1c2-01c8-47f3-8e6c-7b14918b6743-prometheus-ca\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.841210 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.840606 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1db65176-7e26-48d5-b66d-ea279099c362-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.841210 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.840619 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkhzl\" (UniqueName: \"kubernetes.io/projected/1db65176-7e26-48d5-b66d-ea279099c362-kube-api-access-hkhzl\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.841210 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.840633 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/37c7d1c2-01c8-47f3-8e6c-7b14918b6743-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.841210 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.841134 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-9c4wg"] Oct 31 00:39:32.842013 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.841439 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3813ab2d-eb02-48d7-85fe-6c85d327f09e-prometheus-data-science-monitoringstack-rulefiles-1" pod "3813ab2d-eb02-48d7-85fe-6c85d327f09e" (UID: "3813ab2d-eb02-48d7-85fe-6c85d327f09e"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:39:32.843044 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.843010 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ed80a10-b519-4d87-a4a4-0783c84af9aa-otc-internal" pod "5ed80a10-b519-4d87-a4a4-0783c84af9aa" (UID: "5ed80a10-b519-4d87-a4a4-0783c84af9aa"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:39:32.843544 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.843267 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3813ab2d-eb02-48d7-85fe-6c85d327f09e-prometheus-data-science-monitoringstack-rulefiles-0" pod "3813ab2d-eb02-48d7-85fe-6c85d327f09e" (UID: "3813ab2d-eb02-48d7-85fe-6c85d327f09e"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:39:32.844844 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.844716 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3813ab2d-eb02-48d7-85fe-6c85d327f09e-prometheus-data-science-monitoringstack-rulefiles-2" pod "3813ab2d-eb02-48d7-85fe-6c85d327f09e" (UID: "3813ab2d-eb02-48d7-85fe-6c85d327f09e"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:39:32.845537 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.845506 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ed80a10-b519-4d87-a4a4-0783c84af9aa-data-science-collector-ta-client-cert" pod "5ed80a10-b519-4d87-a4a4-0783c84af9aa" (UID: "5ed80a10-b519-4d87-a4a4-0783c84af9aa"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:32.846120 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.846092 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "3813ab2d-eb02-48d7-85fe-6c85d327f09e" (UID: "3813ab2d-eb02-48d7-85fe-6c85d327f09e"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:32.846219 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.846179 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3813ab2d-eb02-48d7-85fe-6c85d327f09e-kube-api-access-vs2n4" pod "3813ab2d-eb02-48d7-85fe-6c85d327f09e" (UID: "3813ab2d-eb02-48d7-85fe-6c85d327f09e"). InnerVolumeSpecName "kube-api-access-vs2n4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:39:32.847065 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.847019 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ed80a10-b519-4d87-a4a4-0783c84af9aa-kube-api-access-8cfsd" pod "5ed80a10-b519-4d87-a4a4-0783c84af9aa" (UID: "5ed80a10-b519-4d87-a4a4-0783c84af9aa"). InnerVolumeSpecName "kube-api-access-8cfsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:39:32.847065 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.847021 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "3813ab2d-eb02-48d7-85fe-6c85d327f09e" (UID: "3813ab2d-eb02-48d7-85fe-6c85d327f09e"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:32.847065 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.847042 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ed80a10-b519-4d87-a4a4-0783c84af9aa-otlp-tls-certs" pod "5ed80a10-b519-4d87-a4a4-0783c84af9aa" (UID: "5ed80a10-b519-4d87-a4a4-0783c84af9aa"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:32.847390 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.847355 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3813ab2d-eb02-48d7-85fe-6c85d327f09e-config-out" pod "3813ab2d-eb02-48d7-85fe-6c85d327f09e" (UID: "3813ab2d-eb02-48d7-85fe-6c85d327f09e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:39:32.847614 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.847528 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-config" pod "3813ab2d-eb02-48d7-85fe-6c85d327f09e" (UID: "3813ab2d-eb02-48d7-85fe-6c85d327f09e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:32.847614 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.847593 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-thanos-prometheus-http-client-file" pod "3813ab2d-eb02-48d7-85fe-6c85d327f09e" (UID: "3813ab2d-eb02-48d7-85fe-6c85d327f09e"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:32.847758 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.847613 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3813ab2d-eb02-48d7-85fe-6c85d327f09e-tls-assets" pod "3813ab2d-eb02-48d7-85fe-6c85d327f09e" (UID: "3813ab2d-eb02-48d7-85fe-6c85d327f09e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:39:32.847758 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.847680 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-secret-prometheus-web-tls-ca" pod "3813ab2d-eb02-48d7-85fe-6c85d327f09e" (UID: "3813ab2d-eb02-48d7-85fe-6c85d327f09e"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:32.848799 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.848775 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ed80a10-b519-4d87-a4a4-0783c84af9aa-monitoring-tls-certs" pod "5ed80a10-b519-4d87-a4a4-0783c84af9aa" (UID: "5ed80a10-b519-4d87-a4a4-0783c84af9aa"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:32.849411 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.849377 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ed80a10-b519-4d87-a4a4-0783c84af9aa-tls-certs" pod "5ed80a10-b519-4d87-a4a4-0783c84af9aa" (UID: "5ed80a10-b519-4d87-a4a4-0783c84af9aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:32.849513 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.849457 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1" pod "3813ab2d-eb02-48d7-85fe-6c85d327f09e" (UID: "3813ab2d-eb02-48d7-85fe-6c85d327f09e"). InnerVolumeSpecName "pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:39:32.855118 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.855097 5274 scope.go:122] "RemoveContainer" containerID="eca7b05bc70871dd2d81521c384c0ab583ab92056a90b67fb595bde33307111a" Oct 31 00:39:32.859472 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.859449 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-web-config" pod "3813ab2d-eb02-48d7-85fe-6c85d327f09e" (UID: "3813ab2d-eb02-48d7-85fe-6c85d327f09e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:32.860001 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.859978 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jrq79"] Oct 31 00:39:32.864573 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.864536 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-jrq79"] Oct 31 00:39:32.864623 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.864609 5274 scope.go:122] "RemoveContainer" containerID="ef15a038a53758d8a9b451b846c4b280ca74720295209ee52b54205e5cce9e69" Oct 31 00:39:32.864908 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:39:32.864890 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef15a038a53758d8a9b451b846c4b280ca74720295209ee52b54205e5cce9e69\": container with ID starting with ef15a038a53758d8a9b451b846c4b280ca74720295209ee52b54205e5cce9e69 not found: ID does not exist" containerID="ef15a038a53758d8a9b451b846c4b280ca74720295209ee52b54205e5cce9e69" Oct 31 00:39:32.864983 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.864915 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef15a038a53758d8a9b451b846c4b280ca74720295209ee52b54205e5cce9e69"} err="failed to get container status \"ef15a038a53758d8a9b451b846c4b280ca74720295209ee52b54205e5cce9e69\": rpc error: code = NotFound desc = could not find container \"ef15a038a53758d8a9b451b846c4b280ca74720295209ee52b54205e5cce9e69\": container with ID starting with ef15a038a53758d8a9b451b846c4b280ca74720295209ee52b54205e5cce9e69 not found: ID does not exist" Oct 31 00:39:32.864983 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.864931 5274 scope.go:122] "RemoveContainer" containerID="662ba948f1acf0026c92a94a2e58228349dd237df6185d12e4e89b851d766cd9" Oct 31 00:39:32.865188 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:39:32.865170 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"662ba948f1acf0026c92a94a2e58228349dd237df6185d12e4e89b851d766cd9\": container with ID starting with 662ba948f1acf0026c92a94a2e58228349dd237df6185d12e4e89b851d766cd9 not found: ID does not exist" containerID="662ba948f1acf0026c92a94a2e58228349dd237df6185d12e4e89b851d766cd9" Oct 31 00:39:32.865228 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.865192 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"662ba948f1acf0026c92a94a2e58228349dd237df6185d12e4e89b851d766cd9"} err="failed to get container status \"662ba948f1acf0026c92a94a2e58228349dd237df6185d12e4e89b851d766cd9\": rpc error: code = NotFound desc = could not find container \"662ba948f1acf0026c92a94a2e58228349dd237df6185d12e4e89b851d766cd9\": container with ID starting with 662ba948f1acf0026c92a94a2e58228349dd237df6185d12e4e89b851d766cd9 not found: ID does not exist" Oct 31 00:39:32.865228 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.865207 5274 scope.go:122] "RemoveContainer" containerID="f7fbe6f697b6b51dd7028cc27c922b46b0762d5953332c396d05cf4aa8f25c8d" Oct 31 00:39:32.865422 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:39:32.865406 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7fbe6f697b6b51dd7028cc27c922b46b0762d5953332c396d05cf4aa8f25c8d\": container with ID starting with f7fbe6f697b6b51dd7028cc27c922b46b0762d5953332c396d05cf4aa8f25c8d not found: ID does not exist" containerID="f7fbe6f697b6b51dd7028cc27c922b46b0762d5953332c396d05cf4aa8f25c8d" Oct 31 00:39:32.865461 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.865425 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7fbe6f697b6b51dd7028cc27c922b46b0762d5953332c396d05cf4aa8f25c8d"} err="failed to get container status \"f7fbe6f697b6b51dd7028cc27c922b46b0762d5953332c396d05cf4aa8f25c8d\": rpc error: code = NotFound desc = could not find container \"f7fbe6f697b6b51dd7028cc27c922b46b0762d5953332c396d05cf4aa8f25c8d\": container with ID starting with f7fbe6f697b6b51dd7028cc27c922b46b0762d5953332c396d05cf4aa8f25c8d not found: ID does not exist" Oct 31 00:39:32.865461 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.865438 5274 scope.go:122] "RemoveContainer" containerID="eca7b05bc70871dd2d81521c384c0ab583ab92056a90b67fb595bde33307111a" Oct 31 00:39:32.865630 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:39:32.865616 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eca7b05bc70871dd2d81521c384c0ab583ab92056a90b67fb595bde33307111a\": container with ID starting with eca7b05bc70871dd2d81521c384c0ab583ab92056a90b67fb595bde33307111a not found: ID does not exist" containerID="eca7b05bc70871dd2d81521c384c0ab583ab92056a90b67fb595bde33307111a" Oct 31 00:39:32.865669 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.865634 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eca7b05bc70871dd2d81521c384c0ab583ab92056a90b67fb595bde33307111a"} err="failed to get container status \"eca7b05bc70871dd2d81521c384c0ab583ab92056a90b67fb595bde33307111a\": rpc error: code = NotFound desc = could not find container \"eca7b05bc70871dd2d81521c384c0ab583ab92056a90b67fb595bde33307111a\": container with ID starting with eca7b05bc70871dd2d81521c384c0ab583ab92056a90b67fb595bde33307111a not found: ID does not exist" Oct 31 00:39:32.865669 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.865645 5274 scope.go:122] "RemoveContainer" containerID="ef15a038a53758d8a9b451b846c4b280ca74720295209ee52b54205e5cce9e69" Oct 31 00:39:32.865879 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.865864 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef15a038a53758d8a9b451b846c4b280ca74720295209ee52b54205e5cce9e69"} err="failed to get container status \"ef15a038a53758d8a9b451b846c4b280ca74720295209ee52b54205e5cce9e69\": rpc error: code = NotFound desc = could not find container \"ef15a038a53758d8a9b451b846c4b280ca74720295209ee52b54205e5cce9e69\": container with ID starting with ef15a038a53758d8a9b451b846c4b280ca74720295209ee52b54205e5cce9e69 not found: ID does not exist" Oct 31 00:39:32.865879 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.865879 5274 scope.go:122] "RemoveContainer" containerID="662ba948f1acf0026c92a94a2e58228349dd237df6185d12e4e89b851d766cd9" Oct 31 00:39:32.866144 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.866127 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"662ba948f1acf0026c92a94a2e58228349dd237df6185d12e4e89b851d766cd9"} err="failed to get container status \"662ba948f1acf0026c92a94a2e58228349dd237df6185d12e4e89b851d766cd9\": rpc error: code = NotFound desc = could not find container \"662ba948f1acf0026c92a94a2e58228349dd237df6185d12e4e89b851d766cd9\": container with ID starting with 662ba948f1acf0026c92a94a2e58228349dd237df6185d12e4e89b851d766cd9 not found: ID does not exist" Oct 31 00:39:32.866192 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.866144 5274 scope.go:122] "RemoveContainer" containerID="f7fbe6f697b6b51dd7028cc27c922b46b0762d5953332c396d05cf4aa8f25c8d" Oct 31 00:39:32.866369 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.866354 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7fbe6f697b6b51dd7028cc27c922b46b0762d5953332c396d05cf4aa8f25c8d"} err="failed to get container status \"f7fbe6f697b6b51dd7028cc27c922b46b0762d5953332c396d05cf4aa8f25c8d\": rpc error: code = NotFound desc = could not find container \"f7fbe6f697b6b51dd7028cc27c922b46b0762d5953332c396d05cf4aa8f25c8d\": container with ID starting with f7fbe6f697b6b51dd7028cc27c922b46b0762d5953332c396d05cf4aa8f25c8d not found: ID does not exist" Oct 31 00:39:32.866414 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.866369 5274 scope.go:122] "RemoveContainer" containerID="eca7b05bc70871dd2d81521c384c0ab583ab92056a90b67fb595bde33307111a" Oct 31 00:39:32.866581 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.866565 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eca7b05bc70871dd2d81521c384c0ab583ab92056a90b67fb595bde33307111a"} err="failed to get container status \"eca7b05bc70871dd2d81521c384c0ab583ab92056a90b67fb595bde33307111a\": rpc error: code = NotFound desc = could not find container \"eca7b05bc70871dd2d81521c384c0ab583ab92056a90b67fb595bde33307111a\": container with ID starting with eca7b05bc70871dd2d81521c384c0ab583ab92056a90b67fb595bde33307111a not found: ID does not exist" Oct 31 00:39:32.866581 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.866580 5274 scope.go:122] "RemoveContainer" containerID="ef15a038a53758d8a9b451b846c4b280ca74720295209ee52b54205e5cce9e69" Oct 31 00:39:32.866789 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.866767 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef15a038a53758d8a9b451b846c4b280ca74720295209ee52b54205e5cce9e69"} err="failed to get container status \"ef15a038a53758d8a9b451b846c4b280ca74720295209ee52b54205e5cce9e69\": rpc error: code = NotFound desc = could not find container \"ef15a038a53758d8a9b451b846c4b280ca74720295209ee52b54205e5cce9e69\": container with ID starting with ef15a038a53758d8a9b451b846c4b280ca74720295209ee52b54205e5cce9e69 not found: ID does not exist" Oct 31 00:39:32.866789 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.866787 5274 scope.go:122] "RemoveContainer" containerID="662ba948f1acf0026c92a94a2e58228349dd237df6185d12e4e89b851d766cd9" Oct 31 00:39:32.867023 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.867009 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"662ba948f1acf0026c92a94a2e58228349dd237df6185d12e4e89b851d766cd9"} err="failed to get container status \"662ba948f1acf0026c92a94a2e58228349dd237df6185d12e4e89b851d766cd9\": rpc error: code = NotFound desc = could not find container \"662ba948f1acf0026c92a94a2e58228349dd237df6185d12e4e89b851d766cd9\": container with ID starting with 662ba948f1acf0026c92a94a2e58228349dd237df6185d12e4e89b851d766cd9 not found: ID does not exist" Oct 31 00:39:32.867023 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.867022 5274 scope.go:122] "RemoveContainer" containerID="f7fbe6f697b6b51dd7028cc27c922b46b0762d5953332c396d05cf4aa8f25c8d" Oct 31 00:39:32.867236 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.867219 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7fbe6f697b6b51dd7028cc27c922b46b0762d5953332c396d05cf4aa8f25c8d"} err="failed to get container status \"f7fbe6f697b6b51dd7028cc27c922b46b0762d5953332c396d05cf4aa8f25c8d\": rpc error: code = NotFound desc = could not find container \"f7fbe6f697b6b51dd7028cc27c922b46b0762d5953332c396d05cf4aa8f25c8d\": container with ID starting with f7fbe6f697b6b51dd7028cc27c922b46b0762d5953332c396d05cf4aa8f25c8d not found: ID does not exist" Oct 31 00:39:32.867236 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.867236 5274 scope.go:122] "RemoveContainer" containerID="eca7b05bc70871dd2d81521c384c0ab583ab92056a90b67fb595bde33307111a" Oct 31 00:39:32.867429 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.867414 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eca7b05bc70871dd2d81521c384c0ab583ab92056a90b67fb595bde33307111a"} err="failed to get container status \"eca7b05bc70871dd2d81521c384c0ab583ab92056a90b67fb595bde33307111a\": rpc error: code = NotFound desc = could not find container \"eca7b05bc70871dd2d81521c384c0ab583ab92056a90b67fb595bde33307111a\": container with ID starting with eca7b05bc70871dd2d81521c384c0ab583ab92056a90b67fb595bde33307111a not found: ID does not exist" Oct 31 00:39:32.867429 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.867428 5274 scope.go:122] "RemoveContainer" containerID="4fd48a1469c39f8446954a1fb2d2408b628676ff83dcbfce418b835c6967278f" Oct 31 00:39:32.874779 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.874752 5274 scope.go:122] "RemoveContainer" containerID="4fd48a1469c39f8446954a1fb2d2408b628676ff83dcbfce418b835c6967278f" Oct 31 00:39:32.875030 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:39:32.875011 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fd48a1469c39f8446954a1fb2d2408b628676ff83dcbfce418b835c6967278f\": container with ID starting with 4fd48a1469c39f8446954a1fb2d2408b628676ff83dcbfce418b835c6967278f not found: ID does not exist" containerID="4fd48a1469c39f8446954a1fb2d2408b628676ff83dcbfce418b835c6967278f" Oct 31 00:39:32.875077 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.875034 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fd48a1469c39f8446954a1fb2d2408b628676ff83dcbfce418b835c6967278f"} err="failed to get container status \"4fd48a1469c39f8446954a1fb2d2408b628676ff83dcbfce418b835c6967278f\": rpc error: code = NotFound desc = could not find container \"4fd48a1469c39f8446954a1fb2d2408b628676ff83dcbfce418b835c6967278f\": container with ID starting with 4fd48a1469c39f8446954a1fb2d2408b628676ff83dcbfce418b835c6967278f not found: ID does not exist" Oct 31 00:39:32.941174 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.941153 5274 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ed80a10-b519-4d87-a4a4-0783c84af9aa-monitoring-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.941174 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.941172 5274 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ed80a10-b519-4d87-a4a4-0783c84af9aa-otlp-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.941304 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.941182 5274 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5ed80a10-b519-4d87-a4a4-0783c84af9aa-data-science-collector-ta-client-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.941304 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.941192 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3813ab2d-eb02-48d7-85fe-6c85d327f09e-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.941304 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.941201 5274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ed80a10-b519-4d87-a4a4-0783c84af9aa-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.941304 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.941211 5274 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-secret-prometheus-web-tls-ca\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.941304 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.941219 5274 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-web-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.941304 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.941229 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3813ab2d-eb02-48d7-85fe-6c85d327f09e-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.941304 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.941238 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3813ab2d-eb02-48d7-85fe-6c85d327f09e-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.941304 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.941247 5274 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3813ab2d-eb02-48d7-85fe-6c85d327f09e-config-out\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.941304 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.941255 5274 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3813ab2d-eb02-48d7-85fe-6c85d327f09e-tls-assets\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.941304 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.941264 5274 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.941304 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.941274 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cfsd\" (UniqueName: \"kubernetes.io/projected/5ed80a10-b519-4d87-a4a4-0783c84af9aa-kube-api-access-8cfsd\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.941304 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.941282 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs2n4\" (UniqueName: \"kubernetes.io/projected/3813ab2d-eb02-48d7-85fe-6c85d327f09e-kube-api-access-vs2n4\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.941304 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.941290 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.941304 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.941300 5274 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.941760 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.941324 5274 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" " Oct 31 00:39:32.941760 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.941333 5274 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5ed80a10-b519-4d87-a4a4-0783c84af9aa-otc-internal\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.941760 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.941341 5274 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/3813ab2d-eb02-48d7-85fe-6c85d327f09e-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.952149 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:32.952135 5274 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1") on node "ip-10-0-129-70.us-east-2.compute.internal" Oct 31 00:39:33.042273 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:33.042241 5274 reconciler_common.go:299] "Volume detached for volume \"pvc-5ada4465-2672-4ad2-9aee-dd3398f6bd70\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fbc4f875a460f1a1\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"csi-51e8027f163b5f98040faf3f8c30848dd44a7a90ca2bd0af2446ea6398ed018d\"" Oct 31 00:39:33.129995 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:33.129968 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:39:33.134327 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:33.134304 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:39:33.149047 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:33.149027 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:39:33.153302 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:33.153280 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 31 00:39:33.481426 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:33.481338 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1db65176-7e26-48d5-b66d-ea279099c362" path="/var/lib/kubelet/pods/1db65176-7e26-48d5-b66d-ea279099c362/volumes" Oct 31 00:39:33.481815 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:33.481793 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37c7d1c2-01c8-47f3-8e6c-7b14918b6743" path="/var/lib/kubelet/pods/37c7d1c2-01c8-47f3-8e6c-7b14918b6743/volumes" Oct 31 00:39:33.482316 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:33.482296 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3813ab2d-eb02-48d7-85fe-6c85d327f09e" path="/var/lib/kubelet/pods/3813ab2d-eb02-48d7-85fe-6c85d327f09e/volumes" Oct 31 00:39:33.483140 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:33.483121 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ed80a10-b519-4d87-a4a4-0783c84af9aa" path="/var/lib/kubelet/pods/5ed80a10-b519-4d87-a4a4-0783c84af9aa/volumes" Oct 31 00:39:34.406420 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:34.406381 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="c815941f-4f86-418a-8d60-e508abc538eb" containerName="alertmanager" probeResult="failure" output="Get \"http://10.134.0.102:9093/-/ready\": dial tcp 10.134.0.102:9093: connect: connection refused" Oct 31 00:39:39.406062 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:39.406022 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="c815941f-4f86-418a-8d60-e508abc538eb" containerName="alertmanager" probeResult="failure" output="Get \"http://10.134.0.102:9093/-/ready\": dial tcp 10.134.0.102:9093: connect: connection refused" Oct 31 00:39:42.535646 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:42.535619 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:39:42.712973 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:42.712933 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c815941f-4f86-418a-8d60-e508abc538eb-kube-api-access-jqrlz\" (UniqueName: \"kubernetes.io/projected/c815941f-4f86-418a-8d60-e508abc538eb-kube-api-access-jqrlz\") pod \"c815941f-4f86-418a-8d60-e508abc538eb\" (UID: \"c815941f-4f86-418a-8d60-e508abc538eb\") " Oct 31 00:39:42.713121 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:42.713016 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c815941f-4f86-418a-8d60-e508abc538eb-web-config\" (UniqueName: \"kubernetes.io/secret/c815941f-4f86-418a-8d60-e508abc538eb-web-config\") pod \"c815941f-4f86-418a-8d60-e508abc538eb\" (UID: \"c815941f-4f86-418a-8d60-e508abc538eb\") " Oct 31 00:39:42.713121 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:42.713051 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c815941f-4f86-418a-8d60-e508abc538eb-tls-assets\" (UniqueName: \"kubernetes.io/projected/c815941f-4f86-418a-8d60-e508abc538eb-tls-assets\") pod \"c815941f-4f86-418a-8d60-e508abc538eb\" (UID: \"c815941f-4f86-418a-8d60-e508abc538eb\") " Oct 31 00:39:42.713121 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:42.713095 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c815941f-4f86-418a-8d60-e508abc538eb-config-volume\" (UniqueName: \"kubernetes.io/secret/c815941f-4f86-418a-8d60-e508abc538eb-config-volume\") pod \"c815941f-4f86-418a-8d60-e508abc538eb\" (UID: \"c815941f-4f86-418a-8d60-e508abc538eb\") " Oct 31 00:39:42.713299 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:42.713125 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c815941f-4f86-418a-8d60-e508abc538eb-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/c815941f-4f86-418a-8d60-e508abc538eb-alertmanager-data-science-monitoringstack-db\") pod \"c815941f-4f86-418a-8d60-e508abc538eb\" (UID: \"c815941f-4f86-418a-8d60-e508abc538eb\") " Oct 31 00:39:42.713299 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:42.713161 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c815941f-4f86-418a-8d60-e508abc538eb-config-out\" (UniqueName: \"kubernetes.io/empty-dir/c815941f-4f86-418a-8d60-e508abc538eb-config-out\") pod \"c815941f-4f86-418a-8d60-e508abc538eb\" (UID: \"c815941f-4f86-418a-8d60-e508abc538eb\") " Oct 31 00:39:42.713299 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:42.713250 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c815941f-4f86-418a-8d60-e508abc538eb-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c815941f-4f86-418a-8d60-e508abc538eb-cluster-tls-config\") pod \"c815941f-4f86-418a-8d60-e508abc538eb\" (UID: \"c815941f-4f86-418a-8d60-e508abc538eb\") " Oct 31 00:39:42.713563 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:42.713518 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c815941f-4f86-418a-8d60-e508abc538eb-alertmanager-data-science-monitoringstack-db" pod "c815941f-4f86-418a-8d60-e508abc538eb" (UID: "c815941f-4f86-418a-8d60-e508abc538eb"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:39:42.715363 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:42.715336 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c815941f-4f86-418a-8d60-e508abc538eb-kube-api-access-jqrlz" pod "c815941f-4f86-418a-8d60-e508abc538eb" (UID: "c815941f-4f86-418a-8d60-e508abc538eb"). InnerVolumeSpecName "kube-api-access-jqrlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:39:42.715645 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:42.715614 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c815941f-4f86-418a-8d60-e508abc538eb-tls-assets" pod "c815941f-4f86-418a-8d60-e508abc538eb" (UID: "c815941f-4f86-418a-8d60-e508abc538eb"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:39:42.715742 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:42.715656 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c815941f-4f86-418a-8d60-e508abc538eb-config-volume" pod "c815941f-4f86-418a-8d60-e508abc538eb" (UID: "c815941f-4f86-418a-8d60-e508abc538eb"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:42.715742 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:42.715725 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c815941f-4f86-418a-8d60-e508abc538eb-config-out" pod "c815941f-4f86-418a-8d60-e508abc538eb" (UID: "c815941f-4f86-418a-8d60-e508abc538eb"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:39:42.720568 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:42.720547 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c815941f-4f86-418a-8d60-e508abc538eb-cluster-tls-config" pod "c815941f-4f86-418a-8d60-e508abc538eb" (UID: "c815941f-4f86-418a-8d60-e508abc538eb"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:42.726365 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:42.726344 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c815941f-4f86-418a-8d60-e508abc538eb-web-config" pod "c815941f-4f86-418a-8d60-e508abc538eb" (UID: "c815941f-4f86-418a-8d60-e508abc538eb"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:42.813828 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:42.813806 5274 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c815941f-4f86-418a-8d60-e508abc538eb-config-out\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:42.813828 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:42.813829 5274 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c815941f-4f86-418a-8d60-e508abc538eb-cluster-tls-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:42.813987 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:42.813839 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqrlz\" (UniqueName: \"kubernetes.io/projected/c815941f-4f86-418a-8d60-e508abc538eb-kube-api-access-jqrlz\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:42.813987 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:42.813852 5274 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c815941f-4f86-418a-8d60-e508abc538eb-web-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:42.813987 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:42.813861 5274 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c815941f-4f86-418a-8d60-e508abc538eb-tls-assets\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:42.813987 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:42.813869 5274 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c815941f-4f86-418a-8d60-e508abc538eb-config-volume\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:42.813987 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:42.813878 5274 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/c815941f-4f86-418a-8d60-e508abc538eb-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:42.867236 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:42.867207 5274 generic.go:356] "Generic (PLEG): container finished" podID="c815941f-4f86-418a-8d60-e508abc538eb" containerID="7087fa10a92db40382436bc890574981812d74c122b4e3325b89b8d6a17026ce" exitCode=0 Oct 31 00:39:42.867343 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:42.867263 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:39:42.867343 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:42.867291 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"c815941f-4f86-418a-8d60-e508abc538eb","Type":"ContainerDied","Data":"7087fa10a92db40382436bc890574981812d74c122b4e3325b89b8d6a17026ce"} Oct 31 00:39:42.867428 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:42.867347 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"c815941f-4f86-418a-8d60-e508abc538eb","Type":"ContainerDied","Data":"ccc721cccaace846272f48426d43ef2e0cdaa83df89236974e6ba3c76a91b75a"} Oct 31 00:39:42.867428 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:42.867374 5274 scope.go:122] "RemoveContainer" containerID="e6f9b9d9a577cc0fcd9018f89f56d6092b781a68cc1c1b12ed9e63ac448d9df1" Oct 31 00:39:42.876153 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:42.876138 5274 scope.go:122] "RemoveContainer" containerID="7087fa10a92db40382436bc890574981812d74c122b4e3325b89b8d6a17026ce" Oct 31 00:39:42.883873 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:42.883857 5274 scope.go:122] "RemoveContainer" containerID="47f94b4ba825bd6c27c3475870294df290c0d6be7e4a9cce2935207be4a74368" Oct 31 00:39:42.891576 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:42.891557 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:39:42.891741 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:42.891729 5274 scope.go:122] "RemoveContainer" containerID="e6f9b9d9a577cc0fcd9018f89f56d6092b781a68cc1c1b12ed9e63ac448d9df1" Oct 31 00:39:42.892012 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:39:42.891990 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6f9b9d9a577cc0fcd9018f89f56d6092b781a68cc1c1b12ed9e63ac448d9df1\": container with ID starting with e6f9b9d9a577cc0fcd9018f89f56d6092b781a68cc1c1b12ed9e63ac448d9df1 not found: ID does not exist" containerID="e6f9b9d9a577cc0fcd9018f89f56d6092b781a68cc1c1b12ed9e63ac448d9df1" Oct 31 00:39:42.892110 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:42.892019 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6f9b9d9a577cc0fcd9018f89f56d6092b781a68cc1c1b12ed9e63ac448d9df1"} err="failed to get container status \"e6f9b9d9a577cc0fcd9018f89f56d6092b781a68cc1c1b12ed9e63ac448d9df1\": rpc error: code = NotFound desc = could not find container \"e6f9b9d9a577cc0fcd9018f89f56d6092b781a68cc1c1b12ed9e63ac448d9df1\": container with ID starting with e6f9b9d9a577cc0fcd9018f89f56d6092b781a68cc1c1b12ed9e63ac448d9df1 not found: ID does not exist" Oct 31 00:39:42.892110 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:42.892035 5274 scope.go:122] "RemoveContainer" containerID="7087fa10a92db40382436bc890574981812d74c122b4e3325b89b8d6a17026ce" Oct 31 00:39:42.892320 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:39:42.892298 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7087fa10a92db40382436bc890574981812d74c122b4e3325b89b8d6a17026ce\": container with ID starting with 7087fa10a92db40382436bc890574981812d74c122b4e3325b89b8d6a17026ce not found: ID does not exist" containerID="7087fa10a92db40382436bc890574981812d74c122b4e3325b89b8d6a17026ce" Oct 31 00:39:42.892363 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:42.892327 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7087fa10a92db40382436bc890574981812d74c122b4e3325b89b8d6a17026ce"} err="failed to get container status \"7087fa10a92db40382436bc890574981812d74c122b4e3325b89b8d6a17026ce\": rpc error: code = NotFound desc = could not find container \"7087fa10a92db40382436bc890574981812d74c122b4e3325b89b8d6a17026ce\": container with ID starting with 7087fa10a92db40382436bc890574981812d74c122b4e3325b89b8d6a17026ce not found: ID does not exist" Oct 31 00:39:42.892363 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:42.892342 5274 scope.go:122] "RemoveContainer" containerID="47f94b4ba825bd6c27c3475870294df290c0d6be7e4a9cce2935207be4a74368" Oct 31 00:39:42.892587 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:39:42.892573 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47f94b4ba825bd6c27c3475870294df290c0d6be7e4a9cce2935207be4a74368\": container with ID starting with 47f94b4ba825bd6c27c3475870294df290c0d6be7e4a9cce2935207be4a74368 not found: ID does not exist" containerID="47f94b4ba825bd6c27c3475870294df290c0d6be7e4a9cce2935207be4a74368" Oct 31 00:39:42.892627 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:42.892589 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47f94b4ba825bd6c27c3475870294df290c0d6be7e4a9cce2935207be4a74368"} err="failed to get container status \"47f94b4ba825bd6c27c3475870294df290c0d6be7e4a9cce2935207be4a74368\": rpc error: code = NotFound desc = could not find container \"47f94b4ba825bd6c27c3475870294df290c0d6be7e4a9cce2935207be4a74368\": container with ID starting with 47f94b4ba825bd6c27c3475870294df290c0d6be7e4a9cce2935207be4a74368 not found: ID does not exist" Oct 31 00:39:42.897978 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:42.897929 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:39:43.480863 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:43.480828 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c815941f-4f86-418a-8d60-e508abc538eb" path="/var/lib/kubelet/pods/c815941f-4f86-418a-8d60-e508abc538eb/volumes" Oct 31 00:39:50.899358 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:39:50.899311 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:39:53.783176 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.783137 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f"] Oct 31 00:39:53.783778 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.783615 5274 cpu_manager.go:414] "containerMap: removing container" podUID="3813ab2d-eb02-48d7-85fe-6c85d327f09e" containerName="init-config-reloader" Oct 31 00:39:53.783778 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.783635 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3813ab2d-eb02-48d7-85fe-6c85d327f09e" containerName="init-config-reloader" Oct 31 00:39:53.783778 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.783663 5274 cpu_manager.go:414] "containerMap: removing container" podUID="c815941f-4f86-418a-8d60-e508abc538eb" containerName="config-reloader" Oct 31 00:39:53.783778 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.783670 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c815941f-4f86-418a-8d60-e508abc538eb" containerName="config-reloader" Oct 31 00:39:53.783778 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.783679 5274 cpu_manager.go:414] "containerMap: removing container" podUID="3813ab2d-eb02-48d7-85fe-6c85d327f09e" containerName="prometheus" Oct 31 00:39:53.783778 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.783686 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3813ab2d-eb02-48d7-85fe-6c85d327f09e" containerName="prometheus" Oct 31 00:39:53.783778 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.783697 5274 cpu_manager.go:414] "containerMap: removing container" podUID="3813ab2d-eb02-48d7-85fe-6c85d327f09e" containerName="config-reloader" Oct 31 00:39:53.783778 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.783704 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3813ab2d-eb02-48d7-85fe-6c85d327f09e" containerName="config-reloader" Oct 31 00:39:53.783778 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.783724 5274 cpu_manager.go:414] "containerMap: removing container" podUID="3813ab2d-eb02-48d7-85fe-6c85d327f09e" containerName="thanos-sidecar" Oct 31 00:39:53.783778 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.783731 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3813ab2d-eb02-48d7-85fe-6c85d327f09e" containerName="thanos-sidecar" Oct 31 00:39:53.783778 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.783744 5274 cpu_manager.go:414] "containerMap: removing container" podUID="c815941f-4f86-418a-8d60-e508abc538eb" containerName="alertmanager" Oct 31 00:39:53.783778 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.783751 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c815941f-4f86-418a-8d60-e508abc538eb" containerName="alertmanager" Oct 31 00:39:53.783778 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.783763 5274 cpu_manager.go:414] "containerMap: removing container" podUID="5ed80a10-b519-4d87-a4a4-0783c84af9aa" containerName="otc-container" Oct 31 00:39:53.783778 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.783769 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ed80a10-b519-4d87-a4a4-0783c84af9aa" containerName="otc-container" Oct 31 00:39:53.784510 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.783791 5274 cpu_manager.go:414] "containerMap: removing container" podUID="1db65176-7e26-48d5-b66d-ea279099c362" containerName="korrel8r" Oct 31 00:39:53.784510 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.783798 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1db65176-7e26-48d5-b66d-ea279099c362" containerName="korrel8r" Oct 31 00:39:53.784510 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.783807 5274 cpu_manager.go:414] "containerMap: removing container" podUID="37c7d1c2-01c8-47f3-8e6c-7b14918b6743" containerName="kube-rbac-proxy" Oct 31 00:39:53.784510 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.783814 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37c7d1c2-01c8-47f3-8e6c-7b14918b6743" containerName="kube-rbac-proxy" Oct 31 00:39:53.784510 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.783823 5274 cpu_manager.go:414] "containerMap: removing container" podUID="c815941f-4f86-418a-8d60-e508abc538eb" containerName="init-config-reloader" Oct 31 00:39:53.784510 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.783829 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c815941f-4f86-418a-8d60-e508abc538eb" containerName="init-config-reloader" Oct 31 00:39:53.784510 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.783921 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="1db65176-7e26-48d5-b66d-ea279099c362" containerName="korrel8r" Oct 31 00:39:53.784510 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.783934 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="37c7d1c2-01c8-47f3-8e6c-7b14918b6743" containerName="kube-rbac-proxy" Oct 31 00:39:53.784510 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.783947 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="c815941f-4f86-418a-8d60-e508abc538eb" containerName="config-reloader" Oct 31 00:39:53.784510 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.783972 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="3813ab2d-eb02-48d7-85fe-6c85d327f09e" containerName="prometheus" Oct 31 00:39:53.784510 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.783980 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="3813ab2d-eb02-48d7-85fe-6c85d327f09e" containerName="config-reloader" Oct 31 00:39:53.784510 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.783989 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="3813ab2d-eb02-48d7-85fe-6c85d327f09e" containerName="thanos-sidecar" Oct 31 00:39:53.784510 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.783998 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="c815941f-4f86-418a-8d60-e508abc538eb" containerName="alertmanager" Oct 31 00:39:53.784510 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.784008 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ed80a10-b519-4d87-a4a4-0783c84af9aa" containerName="otc-container" Oct 31 00:39:53.787473 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.787452 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" Oct 31 00:39:53.795199 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.795175 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway-ca-bundle\"" Oct 31 00:39:53.795306 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.795175 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:39:53.795306 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.795232 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:39:53.795306 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.795240 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway\"" Oct 31 00:39:53.795761 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.795739 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-ca-bundle\"" Oct 31 00:39:53.796550 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.796528 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-data-science-usage-logs-lokistack-gateway-client-http\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f\" (UID: \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" Oct 31 00:39:53.796664 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.796565 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-tenants\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f\" (UID: \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" Oct 31 00:39:53.796664 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.796590 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-tls-secret\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f\" (UID: \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" Oct 31 00:39:53.796779 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.796703 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f\" (UID: \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" Oct 31 00:39:53.796779 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.796756 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9dgx\" (UniqueName: \"kubernetes.io/projected/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-kube-api-access-h9dgx\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f\" (UID: \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" Oct 31 00:39:53.796889 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.796799 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-rbac\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f\" (UID: \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" Oct 31 00:39:53.796889 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.796862 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f\" (UID: \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" Oct 31 00:39:53.796984 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.796913 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-lokistack-gateway\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f\" (UID: \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" Oct 31 00:39:53.800714 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.800688 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f"] Oct 31 00:39:53.812403 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.812376 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway-client-http\"" Oct 31 00:39:53.812518 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.812482 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway-http\"" Oct 31 00:39:53.812584 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.812550 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway\"" Oct 31 00:39:53.898002 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.897977 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-data-science-usage-logs-lokistack-gateway-client-http\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f\" (UID: \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" Oct 31 00:39:53.898131 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.898015 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-tenants\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f\" (UID: \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" Oct 31 00:39:53.898131 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.898038 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-tls-secret\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f\" (UID: \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" Oct 31 00:39:53.898131 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.898080 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f\" (UID: \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" Oct 31 00:39:53.898131 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.898128 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9dgx\" (UniqueName: \"kubernetes.io/projected/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-kube-api-access-h9dgx\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f\" (UID: \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" Oct 31 00:39:53.898356 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.898166 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-rbac\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f\" (UID: \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" Oct 31 00:39:53.898356 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.898198 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f\" (UID: \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" Oct 31 00:39:53.898356 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.898235 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-lokistack-gateway\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f\" (UID: \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" Oct 31 00:39:53.899071 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.899038 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-lokistack-gateway\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f\" (UID: \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" Oct 31 00:39:53.899167 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.899104 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f\" (UID: \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" Oct 31 00:39:53.899629 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.899583 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-rbac\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f\" (UID: \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" Oct 31 00:39:53.900677 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.900652 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-data-science-usage-logs-lokistack-gateway-client-http\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f\" (UID: \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" Oct 31 00:39:53.900777 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.900762 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-tls-secret\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f\" (UID: \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" Oct 31 00:39:53.900874 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.900859 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-tenants\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f\" (UID: \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" Oct 31 00:39:53.901006 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.900984 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f\" (UID: \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" Oct 31 00:39:53.911766 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:53.911746 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9dgx\" (UniqueName: \"kubernetes.io/projected/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-kube-api-access-h9dgx\") pod \"data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f\" (UID: \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" Oct 31 00:39:54.103353 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:54.103276 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" Oct 31 00:39:54.380739 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:54.380716 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f"] Oct 31 00:39:54.383296 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:39:54.383266 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62c0bd2d_d4e2_49c0_9cf6_79acc1364bb1.slice/crio-1751a2a28dea5eedb105827872756754414e4c8ff1502414430a8d73582152ba WatchSource:0}: Error finding container 1751a2a28dea5eedb105827872756754414e4c8ff1502414430a8d73582152ba: Status 404 returned error can't find the container with id 1751a2a28dea5eedb105827872756754414e4c8ff1502414430a8d73582152ba Oct 31 00:39:54.385091 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:54.385076 5274 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:39:54.929192 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:54.929163 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" event={"ID":"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1","Type":"ContainerStarted","Data":"1751a2a28dea5eedb105827872756754414e4c8ff1502414430a8d73582152ba"} Oct 31 00:39:57.948536 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:57.948501 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" event={"ID":"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1","Type":"ContainerStarted","Data":"5399d3fd294a4dc2f657ae57cd261f7344867a6ef267ddfef5c3f775321d9a31"} Oct 31 00:39:59.960544 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:59.960460 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" event={"ID":"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1","Type":"ContainerStarted","Data":"7ea1134091db4eabca6878ea9f1c3c0d5d7a7b30ed3e2ffceaf2ea6fc149c7dd"} Oct 31 00:39:59.960993 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:59.960735 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" Oct 31 00:39:59.960993 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:59.960776 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" Oct 31 00:39:59.967355 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:59.967333 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" Oct 31 00:39:59.969641 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:59.969623 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" Oct 31 00:39:59.988683 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:39:59.988640 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" podStartSLOduration=1.744183793 podStartE2EDuration="6.988629313s" totalImagesPullingTime="5.24444552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:59.986287428 +0000 UTC m=+2436.623106146" watchObservedRunningTime="2026-10-31 00:39:59.988629313 +0000 UTC m=+2436.625448069" Oct 31 00:40:20.900148 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:40:20.900060 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:40:50.899043 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:40:50.898999 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:41:20.898610 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:41:20.898568 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:41:22.394561 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:22.394523 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-qbkbr"] Oct 31 00:41:22.397825 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:22.397799 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-qbkbr" Oct 31 00:41:22.402946 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:22.402917 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-headless-tls\"" Oct 31 00:41:22.416422 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:22.416382 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-qbkbr"] Oct 31 00:41:22.416595 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:22.416572 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-9d8884b6\"" Oct 31 00:41:22.468706 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:22.468673 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9f88\" (UniqueName: \"kubernetes.io/projected/01db571f-7b53-48c7-bfdb-594b1d8cc86f-kube-api-access-x9f88\") pod \"data-science-usage-logs-collector-5dc8ff949d-qbkbr\" (UID: \"01db571f-7b53-48c7-bfdb-594b1d8cc86f\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-qbkbr" Oct 31 00:41:22.468916 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:22.468720 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/01db571f-7b53-48c7-bfdb-594b1d8cc86f-otc-internal\") pod \"data-science-usage-logs-collector-5dc8ff949d-qbkbr\" (UID: \"01db571f-7b53-48c7-bfdb-594b1d8cc86f\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-qbkbr" Oct 31 00:41:22.468916 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:22.468748 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/01db571f-7b53-48c7-bfdb-594b1d8cc86f-otlp-tls-certs\") pod \"data-science-usage-logs-collector-5dc8ff949d-qbkbr\" (UID: \"01db571f-7b53-48c7-bfdb-594b1d8cc86f\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-qbkbr" Oct 31 00:41:22.569362 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:22.569321 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/01db571f-7b53-48c7-bfdb-594b1d8cc86f-otlp-tls-certs\") pod \"data-science-usage-logs-collector-5dc8ff949d-qbkbr\" (UID: \"01db571f-7b53-48c7-bfdb-594b1d8cc86f\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-qbkbr" Oct 31 00:41:22.569571 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:22.569389 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9f88\" (UniqueName: \"kubernetes.io/projected/01db571f-7b53-48c7-bfdb-594b1d8cc86f-kube-api-access-x9f88\") pod \"data-science-usage-logs-collector-5dc8ff949d-qbkbr\" (UID: \"01db571f-7b53-48c7-bfdb-594b1d8cc86f\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-qbkbr" Oct 31 00:41:22.569571 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:22.569416 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/01db571f-7b53-48c7-bfdb-594b1d8cc86f-otc-internal\") pod \"data-science-usage-logs-collector-5dc8ff949d-qbkbr\" (UID: \"01db571f-7b53-48c7-bfdb-594b1d8cc86f\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-qbkbr" Oct 31 00:41:22.570050 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:22.570024 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/01db571f-7b53-48c7-bfdb-594b1d8cc86f-otc-internal\") pod \"data-science-usage-logs-collector-5dc8ff949d-qbkbr\" (UID: \"01db571f-7b53-48c7-bfdb-594b1d8cc86f\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-qbkbr" Oct 31 00:41:22.571876 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:22.571848 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/01db571f-7b53-48c7-bfdb-594b1d8cc86f-otlp-tls-certs\") pod \"data-science-usage-logs-collector-5dc8ff949d-qbkbr\" (UID: \"01db571f-7b53-48c7-bfdb-594b1d8cc86f\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-qbkbr" Oct 31 00:41:22.592743 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:22.592715 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9f88\" (UniqueName: \"kubernetes.io/projected/01db571f-7b53-48c7-bfdb-594b1d8cc86f-kube-api-access-x9f88\") pod \"data-science-usage-logs-collector-5dc8ff949d-qbkbr\" (UID: \"01db571f-7b53-48c7-bfdb-594b1d8cc86f\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-qbkbr" Oct 31 00:41:22.724928 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:22.724840 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-qbkbr" Oct 31 00:41:22.999338 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:22.999311 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-qbkbr"] Oct 31 00:41:23.001931 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:41:23.001904 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01db571f_7b53_48c7_bfdb_594b1d8cc86f.slice/crio-64f583446f68eb6c422b73a3d07dc32e0a5181856ffc046d5fcea3187870d61e WatchSource:0}: Error finding container 64f583446f68eb6c422b73a3d07dc32e0a5181856ffc046d5fcea3187870d61e: Status 404 returned error can't find the container with id 64f583446f68eb6c422b73a3d07dc32e0a5181856ffc046d5fcea3187870d61e Oct 31 00:41:23.004023 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:23.003988 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:41:23.004129 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:23.004047 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:41:23.364293 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:23.364193 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-qbkbr" event={"ID":"01db571f-7b53-48c7-bfdb-594b1d8cc86f","Type":"ContainerStarted","Data":"02f32f9f166bb3202b45b0e8c53c35b305d559c1ac4286a9212612e3a935d732"} Oct 31 00:41:23.364293 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:23.364250 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-qbkbr" event={"ID":"01db571f-7b53-48c7-bfdb-594b1d8cc86f","Type":"ContainerStarted","Data":"64f583446f68eb6c422b73a3d07dc32e0a5181856ffc046d5fcea3187870d61e"} Oct 31 00:41:23.384705 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:23.384657 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-qbkbr" podStartSLOduration=1.384644178 podStartE2EDuration="1.384644178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:23.382120734 +0000 UTC m=+2520.018939460" watchObservedRunningTime="2026-10-31 00:41:23.384644178 +0000 UTC m=+2520.021462894" Oct 31 00:41:25.984786 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:25.984753 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5f5775896-fc8b7"] Oct 31 00:41:25.988142 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:25.988126 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5f5775896-fc8b7" Oct 31 00:41:25.993852 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:25.993829 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 31 00:41:25.997385 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:25.997361 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 31 00:41:25.997543 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:25.997523 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-skh5r\"" Oct 31 00:41:25.998972 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:25.998941 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5f5775896-fc8b7"] Oct 31 00:41:26.098141 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:26.098108 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x79ll\" (UniqueName: \"kubernetes.io/projected/b80e91de-e9e3-4c8f-9791-65c350a2e101-kube-api-access-x79ll\") pod \"korrel8r-5f5775896-fc8b7\" (UID: \"b80e91de-e9e3-4c8f-9791-65c350a2e101\") " pod="redhat-ods-monitoring/korrel8r-5f5775896-fc8b7" Oct 31 00:41:26.098293 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:26.098231 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b80e91de-e9e3-4c8f-9791-65c350a2e101-tls\") pod \"korrel8r-5f5775896-fc8b7\" (UID: \"b80e91de-e9e3-4c8f-9791-65c350a2e101\") " pod="redhat-ods-monitoring/korrel8r-5f5775896-fc8b7" Oct 31 00:41:26.098338 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:26.098307 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b80e91de-e9e3-4c8f-9791-65c350a2e101-config\") pod \"korrel8r-5f5775896-fc8b7\" (UID: \"b80e91de-e9e3-4c8f-9791-65c350a2e101\") " pod="redhat-ods-monitoring/korrel8r-5f5775896-fc8b7" Oct 31 00:41:26.199365 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:26.199331 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b80e91de-e9e3-4c8f-9791-65c350a2e101-config\") pod \"korrel8r-5f5775896-fc8b7\" (UID: \"b80e91de-e9e3-4c8f-9791-65c350a2e101\") " pod="redhat-ods-monitoring/korrel8r-5f5775896-fc8b7" Oct 31 00:41:26.199631 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:26.199376 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x79ll\" (UniqueName: \"kubernetes.io/projected/b80e91de-e9e3-4c8f-9791-65c350a2e101-kube-api-access-x79ll\") pod \"korrel8r-5f5775896-fc8b7\" (UID: \"b80e91de-e9e3-4c8f-9791-65c350a2e101\") " pod="redhat-ods-monitoring/korrel8r-5f5775896-fc8b7" Oct 31 00:41:26.199631 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:26.199430 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b80e91de-e9e3-4c8f-9791-65c350a2e101-tls\") pod \"korrel8r-5f5775896-fc8b7\" (UID: \"b80e91de-e9e3-4c8f-9791-65c350a2e101\") " pod="redhat-ods-monitoring/korrel8r-5f5775896-fc8b7" Oct 31 00:41:26.200086 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:26.200063 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b80e91de-e9e3-4c8f-9791-65c350a2e101-config\") pod \"korrel8r-5f5775896-fc8b7\" (UID: \"b80e91de-e9e3-4c8f-9791-65c350a2e101\") " pod="redhat-ods-monitoring/korrel8r-5f5775896-fc8b7" Oct 31 00:41:26.201886 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:26.201866 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b80e91de-e9e3-4c8f-9791-65c350a2e101-tls\") pod \"korrel8r-5f5775896-fc8b7\" (UID: \"b80e91de-e9e3-4c8f-9791-65c350a2e101\") " pod="redhat-ods-monitoring/korrel8r-5f5775896-fc8b7" Oct 31 00:41:26.212140 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:26.212114 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x79ll\" (UniqueName: \"kubernetes.io/projected/b80e91de-e9e3-4c8f-9791-65c350a2e101-kube-api-access-x79ll\") pod \"korrel8r-5f5775896-fc8b7\" (UID: \"b80e91de-e9e3-4c8f-9791-65c350a2e101\") " pod="redhat-ods-monitoring/korrel8r-5f5775896-fc8b7" Oct 31 00:41:26.262204 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:26.262134 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-74s62"] Oct 31 00:41:26.265527 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:26.265511 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-74s62" Oct 31 00:41:26.275636 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:26.275596 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-dockercfg-sm6jl\"" Oct 31 00:41:26.276985 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:26.276931 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-74s62"] Oct 31 00:41:26.300637 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:26.300609 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f-otlp-tls-certs\") pod \"data-science-usage-logs-collector-778d5f8859-74s62\" (UID: \"95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-74s62" Oct 31 00:41:26.300783 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:26.300696 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5f5775896-fc8b7" Oct 31 00:41:26.300783 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:26.300761 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f-otc-internal\") pod \"data-science-usage-logs-collector-778d5f8859-74s62\" (UID: \"95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-74s62" Oct 31 00:41:26.300924 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:26.300905 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6zsj\" (UniqueName: \"kubernetes.io/projected/95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f-kube-api-access-p6zsj\") pod \"data-science-usage-logs-collector-778d5f8859-74s62\" (UID: \"95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-74s62" Oct 31 00:41:26.401572 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:26.401542 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6zsj\" (UniqueName: \"kubernetes.io/projected/95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f-kube-api-access-p6zsj\") pod \"data-science-usage-logs-collector-778d5f8859-74s62\" (UID: \"95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-74s62" Oct 31 00:41:26.401708 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:26.401584 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f-otlp-tls-certs\") pod \"data-science-usage-logs-collector-778d5f8859-74s62\" (UID: \"95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-74s62" Oct 31 00:41:26.401708 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:26.401616 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f-otc-internal\") pod \"data-science-usage-logs-collector-778d5f8859-74s62\" (UID: \"95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-74s62" Oct 31 00:41:26.402226 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:26.402204 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f-otc-internal\") pod \"data-science-usage-logs-collector-778d5f8859-74s62\" (UID: \"95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-74s62" Oct 31 00:41:26.403855 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:26.403821 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f-otlp-tls-certs\") pod \"data-science-usage-logs-collector-778d5f8859-74s62\" (UID: \"95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-74s62" Oct 31 00:41:26.413971 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:26.413927 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6zsj\" (UniqueName: \"kubernetes.io/projected/95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f-kube-api-access-p6zsj\") pod \"data-science-usage-logs-collector-778d5f8859-74s62\" (UID: \"95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-74s62" Oct 31 00:41:26.577829 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:26.577801 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5f5775896-fc8b7"] Oct 31 00:41:26.579193 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:41:26.579156 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb80e91de_e9e3_4c8f_9791_65c350a2e101.slice/crio-9e1453efcbe727a104c6de4e6724d44cb8e2e39609a715b55692a99212b92d7d WatchSource:0}: Error finding container 9e1453efcbe727a104c6de4e6724d44cb8e2e39609a715b55692a99212b92d7d: Status 404 returned error can't find the container with id 9e1453efcbe727a104c6de4e6724d44cb8e2e39609a715b55692a99212b92d7d Oct 31 00:41:26.579297 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:26.579238 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-74s62" Oct 31 00:41:26.858488 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:26.858465 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-74s62"] Oct 31 00:41:26.860418 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:41:26.860389 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95a57f36_ed6e_4923_a6ac_dd47ea0d5d5f.slice/crio-bd71fae510f0ba6841d8453ed2ec5a424f166243bdbc0c2190c22551ee359d46 WatchSource:0}: Error finding container bd71fae510f0ba6841d8453ed2ec5a424f166243bdbc0c2190c22551ee359d46: Status 404 returned error can't find the container with id bd71fae510f0ba6841d8453ed2ec5a424f166243bdbc0c2190c22551ee359d46 Oct 31 00:41:26.862732 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:26.862701 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:41:26.862813 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:26.862775 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:41:27.386762 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:27.386724 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-74s62" event={"ID":"95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f","Type":"ContainerStarted","Data":"b2874018387d363f0e326e2b804c7caccb6ac005d15eeceabd8740115209f039"} Oct 31 00:41:27.387225 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:27.386789 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-74s62" event={"ID":"95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f","Type":"ContainerStarted","Data":"bd71fae510f0ba6841d8453ed2ec5a424f166243bdbc0c2190c22551ee359d46"} Oct 31 00:41:27.388269 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:27.388247 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5f5775896-fc8b7" event={"ID":"b80e91de-e9e3-4c8f-9791-65c350a2e101","Type":"ContainerStarted","Data":"32dc6711886bec86075a576220e4d17287466df8f897a06ba8355b525acb4eca"} Oct 31 00:41:27.394464 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:27.388687 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5f5775896-fc8b7" event={"ID":"b80e91de-e9e3-4c8f-9791-65c350a2e101","Type":"ContainerStarted","Data":"9e1453efcbe727a104c6de4e6724d44cb8e2e39609a715b55692a99212b92d7d"} Oct 31 00:41:27.394464 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:27.388788 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5f5775896-fc8b7" Oct 31 00:41:27.422314 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:27.422271 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-74s62" podStartSLOduration=1.422259424 podStartE2EDuration="1.422259424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:27.421134449 +0000 UTC m=+2524.057953163" watchObservedRunningTime="2026-10-31 00:41:27.422259424 +0000 UTC m=+2524.059078203" Oct 31 00:41:27.454683 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:27.454630 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5f5775896-fc8b7" podStartSLOduration=2.454613861 podStartE2EDuration="2.454613861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:27.453903377 +0000 UTC m=+2524.090722096" watchObservedRunningTime="2026-10-31 00:41:27.454613861 +0000 UTC m=+2524.091432578" Oct 31 00:41:27.461284 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:27.461258 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-qbkbr"] Oct 31 00:41:27.461466 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:27.461443 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-qbkbr" podUID="01db571f-7b53-48c7-bfdb-594b1d8cc86f" containerName="otc-container" containerID="cri-o://02f32f9f166bb3202b45b0e8c53c35b305d559c1ac4286a9212612e3a935d732" gracePeriod=30 Oct 31 00:41:27.714002 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:27.713975 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-qbkbr" Oct 31 00:41:27.812115 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:27.812081 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01db571f-7b53-48c7-bfdb-594b1d8cc86f-kube-api-access-x9f88\" (UniqueName: \"kubernetes.io/projected/01db571f-7b53-48c7-bfdb-594b1d8cc86f-kube-api-access-x9f88\") pod \"01db571f-7b53-48c7-bfdb-594b1d8cc86f\" (UID: \"01db571f-7b53-48c7-bfdb-594b1d8cc86f\") " Oct 31 00:41:27.812361 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:27.812168 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01db571f-7b53-48c7-bfdb-594b1d8cc86f-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/01db571f-7b53-48c7-bfdb-594b1d8cc86f-otlp-tls-certs\") pod \"01db571f-7b53-48c7-bfdb-594b1d8cc86f\" (UID: \"01db571f-7b53-48c7-bfdb-594b1d8cc86f\") " Oct 31 00:41:27.812361 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:27.812242 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01db571f-7b53-48c7-bfdb-594b1d8cc86f-otc-internal\" (UniqueName: \"kubernetes.io/configmap/01db571f-7b53-48c7-bfdb-594b1d8cc86f-otc-internal\") pod \"01db571f-7b53-48c7-bfdb-594b1d8cc86f\" (UID: \"01db571f-7b53-48c7-bfdb-594b1d8cc86f\") " Oct 31 00:41:27.812867 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:27.812722 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01db571f-7b53-48c7-bfdb-594b1d8cc86f-otc-internal" pod "01db571f-7b53-48c7-bfdb-594b1d8cc86f" (UID: "01db571f-7b53-48c7-bfdb-594b1d8cc86f"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:41:27.813106 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:27.813084 5274 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/01db571f-7b53-48c7-bfdb-594b1d8cc86f-otc-internal\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:27.814553 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:27.814532 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01db571f-7b53-48c7-bfdb-594b1d8cc86f-kube-api-access-x9f88" pod "01db571f-7b53-48c7-bfdb-594b1d8cc86f" (UID: "01db571f-7b53-48c7-bfdb-594b1d8cc86f"). InnerVolumeSpecName "kube-api-access-x9f88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:41:27.814660 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:27.814534 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01db571f-7b53-48c7-bfdb-594b1d8cc86f-otlp-tls-certs" pod "01db571f-7b53-48c7-bfdb-594b1d8cc86f" (UID: "01db571f-7b53-48c7-bfdb-594b1d8cc86f"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:41:27.913755 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:27.913511 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9f88\" (UniqueName: \"kubernetes.io/projected/01db571f-7b53-48c7-bfdb-594b1d8cc86f-kube-api-access-x9f88\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:27.913898 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:27.913771 5274 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/01db571f-7b53-48c7-bfdb-594b1d8cc86f-otlp-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:27.955162 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:27.955069 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb"] Oct 31 00:41:27.955483 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:27.955466 5274 cpu_manager.go:414] "containerMap: removing container" podUID="01db571f-7b53-48c7-bfdb-594b1d8cc86f" containerName="otc-container" Oct 31 00:41:27.955483 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:27.955482 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01db571f-7b53-48c7-bfdb-594b1d8cc86f" containerName="otc-container" Oct 31 00:41:27.955610 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:27.955555 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="01db571f-7b53-48c7-bfdb-594b1d8cc86f" containerName="otc-container" Oct 31 00:41:27.959149 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:27.959127 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" Oct 31 00:41:27.968498 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:27.968472 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-ca-bundle\"" Oct 31 00:41:27.968641 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:27.968627 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ca-bundle\"" Oct 31 00:41:27.968709 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:27.968675 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway\"" Oct 31 00:41:27.974082 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:27.974058 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb"] Oct 31 00:41:27.993985 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:27.993919 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway\"" Oct 31 00:41:27.994228 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:27.994202 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-http\"" Oct 31 00:41:27.994347 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:27.994229 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-client-http\"" Oct 31 00:41:27.994347 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:27.994336 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-dockercfg-whtcj\"" Oct 31 00:41:28.015234 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:28.015202 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-tenants\") pod \"data-science-lokistack-gateway-66cfb97849-fd4tb\" (UID: \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" Oct 31 00:41:28.015360 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:28.015248 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-tls-secret\") pod \"data-science-lokistack-gateway-66cfb97849-fd4tb\" (UID: \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" Oct 31 00:41:28.015403 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:28.015383 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-lokistack-gateway\") pod \"data-science-lokistack-gateway-66cfb97849-fd4tb\" (UID: \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" Oct 31 00:41:28.015439 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:28.015410 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-66cfb97849-fd4tb\" (UID: \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" Oct 31 00:41:28.015491 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:28.015437 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-66cfb97849-fd4tb\" (UID: \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" Oct 31 00:41:28.015491 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:28.015477 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-66cfb97849-fd4tb\" (UID: \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" Oct 31 00:41:28.015561 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:28.015549 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxcn7\" (UniqueName: \"kubernetes.io/projected/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-kube-api-access-wxcn7\") pod \"data-science-lokistack-gateway-66cfb97849-fd4tb\" (UID: \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" Oct 31 00:41:28.015602 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:28.015588 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-rbac\") pod \"data-science-lokistack-gateway-66cfb97849-fd4tb\" (UID: \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" Oct 31 00:41:28.116898 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:28.116861 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-lokistack-gateway\") pod \"data-science-lokistack-gateway-66cfb97849-fd4tb\" (UID: \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" Oct 31 00:41:28.116898 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:28.116898 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-66cfb97849-fd4tb\" (UID: \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" Oct 31 00:41:28.117152 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:28.116919 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-66cfb97849-fd4tb\" (UID: \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" Oct 31 00:41:28.117152 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:28.117022 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-66cfb97849-fd4tb\" (UID: \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" Oct 31 00:41:28.117152 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:28.117048 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxcn7\" (UniqueName: \"kubernetes.io/projected/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-kube-api-access-wxcn7\") pod \"data-science-lokistack-gateway-66cfb97849-fd4tb\" (UID: \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" Oct 31 00:41:28.117152 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:28.117083 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-rbac\") pod \"data-science-lokistack-gateway-66cfb97849-fd4tb\" (UID: \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" Oct 31 00:41:28.117351 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:28.117216 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-tenants\") pod \"data-science-lokistack-gateway-66cfb97849-fd4tb\" (UID: \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" Oct 31 00:41:28.117351 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:28.117282 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-tls-secret\") pod \"data-science-lokistack-gateway-66cfb97849-fd4tb\" (UID: \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" Oct 31 00:41:28.117838 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:28.117815 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-lokistack-gateway\") pod \"data-science-lokistack-gateway-66cfb97849-fd4tb\" (UID: \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" Oct 31 00:41:28.118032 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:28.117844 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-66cfb97849-fd4tb\" (UID: \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" Oct 31 00:41:28.118032 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:28.117918 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-rbac\") pod \"data-science-lokistack-gateway-66cfb97849-fd4tb\" (UID: \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" Oct 31 00:41:28.120191 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:28.120163 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-tenants\") pod \"data-science-lokistack-gateway-66cfb97849-fd4tb\" (UID: \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" Oct 31 00:41:28.120191 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:28.120181 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-66cfb97849-fd4tb\" (UID: \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" Oct 31 00:41:28.120361 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:28.120307 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-66cfb97849-fd4tb\" (UID: \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" Oct 31 00:41:28.120361 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:28.120307 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-tls-secret\") pod \"data-science-lokistack-gateway-66cfb97849-fd4tb\" (UID: \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" Oct 31 00:41:28.131210 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:28.131186 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxcn7\" (UniqueName: \"kubernetes.io/projected/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-kube-api-access-wxcn7\") pod \"data-science-lokistack-gateway-66cfb97849-fd4tb\" (UID: \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" Oct 31 00:41:28.275680 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:28.275590 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" Oct 31 00:41:28.395226 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:28.395186 5274 generic.go:356] "Generic (PLEG): container finished" podID="01db571f-7b53-48c7-bfdb-594b1d8cc86f" containerID="02f32f9f166bb3202b45b0e8c53c35b305d559c1ac4286a9212612e3a935d732" exitCode=0 Oct 31 00:41:28.395226 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:28.395226 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-qbkbr" Oct 31 00:41:28.395742 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:28.395263 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-qbkbr" event={"ID":"01db571f-7b53-48c7-bfdb-594b1d8cc86f","Type":"ContainerDied","Data":"02f32f9f166bb3202b45b0e8c53c35b305d559c1ac4286a9212612e3a935d732"} Oct 31 00:41:28.395742 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:28.395326 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-qbkbr" event={"ID":"01db571f-7b53-48c7-bfdb-594b1d8cc86f","Type":"ContainerDied","Data":"64f583446f68eb6c422b73a3d07dc32e0a5181856ffc046d5fcea3187870d61e"} Oct 31 00:41:28.395742 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:28.395372 5274 scope.go:122] "RemoveContainer" containerID="02f32f9f166bb3202b45b0e8c53c35b305d559c1ac4286a9212612e3a935d732" Oct 31 00:41:28.404180 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:28.404162 5274 scope.go:122] "RemoveContainer" containerID="02f32f9f166bb3202b45b0e8c53c35b305d559c1ac4286a9212612e3a935d732" Oct 31 00:41:28.404437 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:41:28.404418 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02f32f9f166bb3202b45b0e8c53c35b305d559c1ac4286a9212612e3a935d732\": container with ID starting with 02f32f9f166bb3202b45b0e8c53c35b305d559c1ac4286a9212612e3a935d732 not found: ID does not exist" containerID="02f32f9f166bb3202b45b0e8c53c35b305d559c1ac4286a9212612e3a935d732" Oct 31 00:41:28.404502 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:28.404446 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02f32f9f166bb3202b45b0e8c53c35b305d559c1ac4286a9212612e3a935d732"} err="failed to get container status \"02f32f9f166bb3202b45b0e8c53c35b305d559c1ac4286a9212612e3a935d732\": rpc error: code = NotFound desc = could not find container \"02f32f9f166bb3202b45b0e8c53c35b305d559c1ac4286a9212612e3a935d732\": container with ID starting with 02f32f9f166bb3202b45b0e8c53c35b305d559c1ac4286a9212612e3a935d732 not found: ID does not exist" Oct 31 00:41:28.418018 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:28.417989 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-qbkbr"] Oct 31 00:41:28.424358 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:28.424331 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-qbkbr"] Oct 31 00:41:28.558895 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:28.558868 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb"] Oct 31 00:41:28.560970 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:41:28.560927 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2141e01_9d1a_4688_a9b3_cc501d0edbf9.slice/crio-3c278f354b06a05c9180470b1e1cc94d19ce1dfa2f9006a6cd377696814ec2c9 WatchSource:0}: Error finding container 3c278f354b06a05c9180470b1e1cc94d19ce1dfa2f9006a6cd377696814ec2c9: Status 404 returned error can't find the container with id 3c278f354b06a05c9180470b1e1cc94d19ce1dfa2f9006a6cd377696814ec2c9 Oct 31 00:41:29.402445 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:29.402410 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" event={"ID":"c2141e01-9d1a-4688-a9b3-cc501d0edbf9","Type":"ContainerStarted","Data":"1b1998d006e70929ca79a121419451a4dc67905528257fde650b1146aafca7ef"} Oct 31 00:41:29.402836 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:29.402456 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" event={"ID":"c2141e01-9d1a-4688-a9b3-cc501d0edbf9","Type":"ContainerStarted","Data":"e5aa4c74d61285c4a418914d4fd343290dccb087eab52910db959b122993b38e"} Oct 31 00:41:29.402836 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:29.402474 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" event={"ID":"c2141e01-9d1a-4688-a9b3-cc501d0edbf9","Type":"ContainerStarted","Data":"3c278f354b06a05c9180470b1e1cc94d19ce1dfa2f9006a6cd377696814ec2c9"} Oct 31 00:41:29.402836 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:29.402620 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" Oct 31 00:41:29.408532 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:29.408502 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" Oct 31 00:41:29.427659 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:29.427615 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" podStartSLOduration=2.427603596 podStartE2EDuration="2.427603596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:29.427150397 +0000 UTC m=+2526.063969114" watchObservedRunningTime="2026-10-31 00:41:29.427603596 +0000 UTC m=+2526.064422312" Oct 31 00:41:29.484378 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:29.484352 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01db571f-7b53-48c7-bfdb-594b1d8cc86f" path="/var/lib/kubelet/pods/01db571f-7b53-48c7-bfdb-594b1d8cc86f/volumes" Oct 31 00:41:30.407121 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:30.407089 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" Oct 31 00:41:30.414818 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:30.414795 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" Oct 31 00:41:31.698599 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.698564 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-w6w4b"] Oct 31 00:41:31.702870 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.702848 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-w6w4b" Oct 31 00:41:31.709743 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.709720 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-collector-dockercfg-fkhp6\"" Oct 31 00:41:31.712150 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.712122 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-metrics\"" Oct 31 00:41:31.712150 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.712145 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-collector-token\"" Oct 31 00:41:31.713530 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.713512 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"openshift-service-ca.crt\"" Oct 31 00:41:31.713652 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.713582 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-config\"" Oct 31 00:41:31.714866 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.714838 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"kube-root-ca.crt\"" Oct 31 00:41:31.716666 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.716637 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-trustbundle\"" Oct 31 00:41:31.718396 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.718374 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-w6w4b"] Oct 31 00:41:31.746598 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.746574 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/e7c315e6-62d3-4ae5-9875-2bddd67613d0-varlogpods\") pod \"data-science-cluster-log-forwarder-w6w4b\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " pod="openshift-logging/data-science-cluster-log-forwarder-w6w4b" Oct 31 00:41:31.746715 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.746606 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fxc2\" (UniqueName: \"kubernetes.io/projected/e7c315e6-62d3-4ae5-9875-2bddd67613d0-kube-api-access-6fxc2\") pod \"data-science-cluster-log-forwarder-w6w4b\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " pod="openshift-logging/data-science-cluster-log-forwarder-w6w4b" Oct 31 00:41:31.746715 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.746631 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/e7c315e6-62d3-4ae5-9875-2bddd67613d0-config-openshift-service-cacrt\") pod \"data-science-cluster-log-forwarder-w6w4b\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " pod="openshift-logging/data-science-cluster-log-forwarder-w6w4b" Oct 31 00:41:31.746715 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.746683 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7c315e6-62d3-4ae5-9875-2bddd67613d0-trusted-ca\") pod \"data-science-cluster-log-forwarder-w6w4b\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " pod="openshift-logging/data-science-cluster-log-forwarder-w6w4b" Oct 31 00:41:31.746715 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.746712 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/e7c315e6-62d3-4ae5-9875-2bddd67613d0-sa-token\") pod \"data-science-cluster-log-forwarder-w6w4b\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " pod="openshift-logging/data-science-cluster-log-forwarder-w6w4b" Oct 31 00:41:31.746923 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.746733 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/e7c315e6-62d3-4ae5-9875-2bddd67613d0-data-science-cluster-log-forwarder-collector-token\") pod \"data-science-cluster-log-forwarder-w6w4b\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " pod="openshift-logging/data-science-cluster-log-forwarder-w6w4b" Oct 31 00:41:31.746923 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.746855 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/e7c315e6-62d3-4ae5-9875-2bddd67613d0-metrics\") pod \"data-science-cluster-log-forwarder-w6w4b\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " pod="openshift-logging/data-science-cluster-log-forwarder-w6w4b" Oct 31 00:41:31.746923 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.746912 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e7c315e6-62d3-4ae5-9875-2bddd67613d0-config\") pod \"data-science-cluster-log-forwarder-w6w4b\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " pod="openshift-logging/data-science-cluster-log-forwarder-w6w4b" Oct 31 00:41:31.747099 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.746936 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/e7c315e6-62d3-4ae5-9875-2bddd67613d0-datadir\") pod \"data-science-cluster-log-forwarder-w6w4b\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " pod="openshift-logging/data-science-cluster-log-forwarder-w6w4b" Oct 31 00:41:31.747099 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.747032 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7c315e6-62d3-4ae5-9875-2bddd67613d0-tmp\") pod \"data-science-cluster-log-forwarder-w6w4b\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " pod="openshift-logging/data-science-cluster-log-forwarder-w6w4b" Oct 31 00:41:31.747099 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.747060 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/e7c315e6-62d3-4ae5-9875-2bddd67613d0-entrypoint\") pod \"data-science-cluster-log-forwarder-w6w4b\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " pod="openshift-logging/data-science-cluster-log-forwarder-w6w4b" Oct 31 00:41:31.793576 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.793548 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-w6w4b"] Oct 31 00:41:31.793777 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:41:31.793759 5274 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[config config-openshift-service-cacrt data-science-cluster-log-forwarder-collector-token datadir entrypoint kube-api-access-6fxc2 metrics sa-token tmp trusted-ca varlogpods], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-logging/data-science-cluster-log-forwarder-w6w4b" podUID="e7c315e6-62d3-4ae5-9875-2bddd67613d0" Oct 31 00:41:31.847648 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.847620 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/e7c315e6-62d3-4ae5-9875-2bddd67613d0-config-openshift-service-cacrt\") pod \"data-science-cluster-log-forwarder-w6w4b\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " pod="openshift-logging/data-science-cluster-log-forwarder-w6w4b" Oct 31 00:41:31.847761 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.847649 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7c315e6-62d3-4ae5-9875-2bddd67613d0-trusted-ca\") pod \"data-science-cluster-log-forwarder-w6w4b\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " pod="openshift-logging/data-science-cluster-log-forwarder-w6w4b" Oct 31 00:41:31.847761 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.847676 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/e7c315e6-62d3-4ae5-9875-2bddd67613d0-sa-token\") pod \"data-science-cluster-log-forwarder-w6w4b\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " pod="openshift-logging/data-science-cluster-log-forwarder-w6w4b" Oct 31 00:41:31.847761 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.847695 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/e7c315e6-62d3-4ae5-9875-2bddd67613d0-data-science-cluster-log-forwarder-collector-token\") pod \"data-science-cluster-log-forwarder-w6w4b\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " pod="openshift-logging/data-science-cluster-log-forwarder-w6w4b" Oct 31 00:41:31.847761 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.847727 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/e7c315e6-62d3-4ae5-9875-2bddd67613d0-metrics\") pod \"data-science-cluster-log-forwarder-w6w4b\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " pod="openshift-logging/data-science-cluster-log-forwarder-w6w4b" Oct 31 00:41:31.847761 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.847753 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e7c315e6-62d3-4ae5-9875-2bddd67613d0-config\") pod \"data-science-cluster-log-forwarder-w6w4b\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " pod="openshift-logging/data-science-cluster-log-forwarder-w6w4b" Oct 31 00:41:31.848044 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.847768 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/e7c315e6-62d3-4ae5-9875-2bddd67613d0-datadir\") pod \"data-science-cluster-log-forwarder-w6w4b\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " pod="openshift-logging/data-science-cluster-log-forwarder-w6w4b" Oct 31 00:41:31.848044 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.847831 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/e7c315e6-62d3-4ae5-9875-2bddd67613d0-datadir\") pod \"data-science-cluster-log-forwarder-w6w4b\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " pod="openshift-logging/data-science-cluster-log-forwarder-w6w4b" Oct 31 00:41:31.848044 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.848036 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7c315e6-62d3-4ae5-9875-2bddd67613d0-tmp\") pod \"data-science-cluster-log-forwarder-w6w4b\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " pod="openshift-logging/data-science-cluster-log-forwarder-w6w4b" Oct 31 00:41:31.848255 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.848070 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/e7c315e6-62d3-4ae5-9875-2bddd67613d0-entrypoint\") pod \"data-science-cluster-log-forwarder-w6w4b\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " pod="openshift-logging/data-science-cluster-log-forwarder-w6w4b" Oct 31 00:41:31.848255 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.848149 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/e7c315e6-62d3-4ae5-9875-2bddd67613d0-varlogpods\") pod \"data-science-cluster-log-forwarder-w6w4b\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " pod="openshift-logging/data-science-cluster-log-forwarder-w6w4b" Oct 31 00:41:31.848255 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.848196 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fxc2\" (UniqueName: \"kubernetes.io/projected/e7c315e6-62d3-4ae5-9875-2bddd67613d0-kube-api-access-6fxc2\") pod \"data-science-cluster-log-forwarder-w6w4b\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " pod="openshift-logging/data-science-cluster-log-forwarder-w6w4b" Oct 31 00:41:31.848407 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.848373 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/e7c315e6-62d3-4ae5-9875-2bddd67613d0-config-openshift-service-cacrt\") pod \"data-science-cluster-log-forwarder-w6w4b\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " pod="openshift-logging/data-science-cluster-log-forwarder-w6w4b" Oct 31 00:41:31.848461 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.848443 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/e7c315e6-62d3-4ae5-9875-2bddd67613d0-varlogpods\") pod \"data-science-cluster-log-forwarder-w6w4b\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " pod="openshift-logging/data-science-cluster-log-forwarder-w6w4b" Oct 31 00:41:31.848533 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.848509 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7c315e6-62d3-4ae5-9875-2bddd67613d0-trusted-ca\") pod \"data-science-cluster-log-forwarder-w6w4b\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " pod="openshift-logging/data-science-cluster-log-forwarder-w6w4b" Oct 31 00:41:31.848586 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.848530 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e7c315e6-62d3-4ae5-9875-2bddd67613d0-config\") pod \"data-science-cluster-log-forwarder-w6w4b\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " pod="openshift-logging/data-science-cluster-log-forwarder-w6w4b" Oct 31 00:41:31.848796 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.848777 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/e7c315e6-62d3-4ae5-9875-2bddd67613d0-entrypoint\") pod \"data-science-cluster-log-forwarder-w6w4b\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " pod="openshift-logging/data-science-cluster-log-forwarder-w6w4b" Oct 31 00:41:31.850313 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.850293 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/e7c315e6-62d3-4ae5-9875-2bddd67613d0-metrics\") pod \"data-science-cluster-log-forwarder-w6w4b\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " pod="openshift-logging/data-science-cluster-log-forwarder-w6w4b" Oct 31 00:41:31.850405 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.850383 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7c315e6-62d3-4ae5-9875-2bddd67613d0-tmp\") pod \"data-science-cluster-log-forwarder-w6w4b\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " pod="openshift-logging/data-science-cluster-log-forwarder-w6w4b" Oct 31 00:41:31.850452 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.850430 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/e7c315e6-62d3-4ae5-9875-2bddd67613d0-data-science-cluster-log-forwarder-collector-token\") pod \"data-science-cluster-log-forwarder-w6w4b\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " pod="openshift-logging/data-science-cluster-log-forwarder-w6w4b" Oct 31 00:41:31.859769 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.859725 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/e7c315e6-62d3-4ae5-9875-2bddd67613d0-sa-token\") pod \"data-science-cluster-log-forwarder-w6w4b\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " pod="openshift-logging/data-science-cluster-log-forwarder-w6w4b" Oct 31 00:41:31.860179 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:31.860160 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fxc2\" (UniqueName: \"kubernetes.io/projected/e7c315e6-62d3-4ae5-9875-2bddd67613d0-kube-api-access-6fxc2\") pod \"data-science-cluster-log-forwarder-w6w4b\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " pod="openshift-logging/data-science-cluster-log-forwarder-w6w4b" Oct 31 00:41:32.424267 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:32.424238 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-w6w4b" Oct 31 00:41:32.432772 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:32.432747 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-w6w4b" Oct 31 00:41:32.553929 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:32.553899 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/host-path/e7c315e6-62d3-4ae5-9875-2bddd67613d0-datadir\" (UniqueName: \"kubernetes.io/host-path/e7c315e6-62d3-4ae5-9875-2bddd67613d0-datadir\") pod \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " Oct 31 00:41:32.554078 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:32.553946 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7c315e6-62d3-4ae5-9875-2bddd67613d0-sa-token\" (UniqueName: \"kubernetes.io/projected/e7c315e6-62d3-4ae5-9875-2bddd67613d0-sa-token\") pod \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " Oct 31 00:41:32.554078 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:32.553992 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/e7c315e6-62d3-4ae5-9875-2bddd67613d0-datadir" pod "e7c315e6-62d3-4ae5-9875-2bddd67613d0" (UID: "e7c315e6-62d3-4ae5-9875-2bddd67613d0"). InnerVolumeSpecName "datadir". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Oct 31 00:41:32.554078 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:32.554021 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7c315e6-62d3-4ae5-9875-2bddd67613d0-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7c315e6-62d3-4ae5-9875-2bddd67613d0-tmp\") pod \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " Oct 31 00:41:32.554078 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:32.554057 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7c315e6-62d3-4ae5-9875-2bddd67613d0-kube-api-access-6fxc2\" (UniqueName: \"kubernetes.io/projected/e7c315e6-62d3-4ae5-9875-2bddd67613d0-kube-api-access-6fxc2\") pod \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " Oct 31 00:41:32.554078 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:32.554076 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7c315e6-62d3-4ae5-9875-2bddd67613d0-config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/e7c315e6-62d3-4ae5-9875-2bddd67613d0-config-openshift-service-cacrt\") pod \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " Oct 31 00:41:32.554314 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:32.554092 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7c315e6-62d3-4ae5-9875-2bddd67613d0-metrics\" (UniqueName: \"kubernetes.io/secret/e7c315e6-62d3-4ae5-9875-2bddd67613d0-metrics\") pod \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " Oct 31 00:41:32.554314 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:32.554127 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7c315e6-62d3-4ae5-9875-2bddd67613d0-data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/e7c315e6-62d3-4ae5-9875-2bddd67613d0-data-science-cluster-log-forwarder-collector-token\") pod \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " Oct 31 00:41:32.554314 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:32.554157 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7c315e6-62d3-4ae5-9875-2bddd67613d0-config\" (UniqueName: \"kubernetes.io/configmap/e7c315e6-62d3-4ae5-9875-2bddd67613d0-config\") pod \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " Oct 31 00:41:32.554314 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:32.554187 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/host-path/e7c315e6-62d3-4ae5-9875-2bddd67613d0-varlogpods\" (UniqueName: \"kubernetes.io/host-path/e7c315e6-62d3-4ae5-9875-2bddd67613d0-varlogpods\") pod \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " Oct 31 00:41:32.554314 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:32.554216 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7c315e6-62d3-4ae5-9875-2bddd67613d0-entrypoint\" (UniqueName: \"kubernetes.io/configmap/e7c315e6-62d3-4ae5-9875-2bddd67613d0-entrypoint\") pod \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " Oct 31 00:41:32.554314 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:32.554263 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7c315e6-62d3-4ae5-9875-2bddd67613d0-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7c315e6-62d3-4ae5-9875-2bddd67613d0-trusted-ca\") pod \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\" (UID: \"e7c315e6-62d3-4ae5-9875-2bddd67613d0\") " Oct 31 00:41:32.554652 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:32.554520 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/e7c315e6-62d3-4ae5-9875-2bddd67613d0-varlogpods" pod "e7c315e6-62d3-4ae5-9875-2bddd67613d0" (UID: "e7c315e6-62d3-4ae5-9875-2bddd67613d0"). InnerVolumeSpecName "varlogpods". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Oct 31 00:41:32.554652 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:32.554536 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7c315e6-62d3-4ae5-9875-2bddd67613d0-config-openshift-service-cacrt" pod "e7c315e6-62d3-4ae5-9875-2bddd67613d0" (UID: "e7c315e6-62d3-4ae5-9875-2bddd67613d0"). InnerVolumeSpecName "config-openshift-service-cacrt". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:41:32.554782 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:32.554760 5274 reconciler_common.go:299] "Volume detached for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/e7c315e6-62d3-4ae5-9875-2bddd67613d0-config-openshift-service-cacrt\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:32.554840 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:32.554784 5274 reconciler_common.go:299] "Volume detached for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/e7c315e6-62d3-4ae5-9875-2bddd67613d0-varlogpods\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:32.554840 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:32.554803 5274 reconciler_common.go:299] "Volume detached for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/e7c315e6-62d3-4ae5-9875-2bddd67613d0-datadir\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:32.554982 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:32.554877 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7c315e6-62d3-4ae5-9875-2bddd67613d0-entrypoint" pod "e7c315e6-62d3-4ae5-9875-2bddd67613d0" (UID: "e7c315e6-62d3-4ae5-9875-2bddd67613d0"). InnerVolumeSpecName "entrypoint". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:41:32.554982 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:32.554902 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7c315e6-62d3-4ae5-9875-2bddd67613d0-config" pod "e7c315e6-62d3-4ae5-9875-2bddd67613d0" (UID: "e7c315e6-62d3-4ae5-9875-2bddd67613d0"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:41:32.555317 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:32.555292 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7c315e6-62d3-4ae5-9875-2bddd67613d0-trusted-ca" pod "e7c315e6-62d3-4ae5-9875-2bddd67613d0" (UID: "e7c315e6-62d3-4ae5-9875-2bddd67613d0"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:41:32.556495 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:32.556408 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7c315e6-62d3-4ae5-9875-2bddd67613d0-metrics" pod "e7c315e6-62d3-4ae5-9875-2bddd67613d0" (UID: "e7c315e6-62d3-4ae5-9875-2bddd67613d0"). InnerVolumeSpecName "metrics". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:41:32.556495 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:32.556474 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c315e6-62d3-4ae5-9875-2bddd67613d0-tmp" pod "e7c315e6-62d3-4ae5-9875-2bddd67613d0" (UID: "e7c315e6-62d3-4ae5-9875-2bddd67613d0"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:41:32.556621 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:32.556520 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7c315e6-62d3-4ae5-9875-2bddd67613d0-kube-api-access-6fxc2" pod "e7c315e6-62d3-4ae5-9875-2bddd67613d0" (UID: "e7c315e6-62d3-4ae5-9875-2bddd67613d0"). InnerVolumeSpecName "kube-api-access-6fxc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:41:32.556621 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:32.556566 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7c315e6-62d3-4ae5-9875-2bddd67613d0-sa-token" pod "e7c315e6-62d3-4ae5-9875-2bddd67613d0" (UID: "e7c315e6-62d3-4ae5-9875-2bddd67613d0"). InnerVolumeSpecName "sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:41:32.557268 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:32.557246 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7c315e6-62d3-4ae5-9875-2bddd67613d0-data-science-cluster-log-forwarder-collector-token" pod "e7c315e6-62d3-4ae5-9875-2bddd67613d0" (UID: "e7c315e6-62d3-4ae5-9875-2bddd67613d0"). InnerVolumeSpecName "data-science-cluster-log-forwarder-collector-token". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:41:32.655617 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:32.655588 5274 reconciler_common.go:299] "Volume detached for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/e7c315e6-62d3-4ae5-9875-2bddd67613d0-sa-token\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:32.655617 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:32.655617 5274 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7c315e6-62d3-4ae5-9875-2bddd67613d0-tmp\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:32.655617 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:32.655628 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fxc2\" (UniqueName: \"kubernetes.io/projected/e7c315e6-62d3-4ae5-9875-2bddd67613d0-kube-api-access-6fxc2\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:32.655793 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:32.655637 5274 reconciler_common.go:299] "Volume detached for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/e7c315e6-62d3-4ae5-9875-2bddd67613d0-metrics\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:32.655793 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:32.655647 5274 reconciler_common.go:299] "Volume detached for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/e7c315e6-62d3-4ae5-9875-2bddd67613d0-data-science-cluster-log-forwarder-collector-token\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:32.655793 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:32.655655 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e7c315e6-62d3-4ae5-9875-2bddd67613d0-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:32.655793 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:32.655664 5274 reconciler_common.go:299] "Volume detached for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/e7c315e6-62d3-4ae5-9875-2bddd67613d0-entrypoint\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:32.655793 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:32.655671 5274 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7c315e6-62d3-4ae5-9875-2bddd67613d0-trusted-ca\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:33.429552 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.429524 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-w6w4b" Oct 31 00:41:33.466673 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.466643 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-w6w4b"] Oct 31 00:41:33.471141 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.471112 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-w6w4b"] Oct 31 00:41:33.483669 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.483647 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7c315e6-62d3-4ae5-9875-2bddd67613d0" path="/var/lib/kubelet/pods/e7c315e6-62d3-4ae5-9875-2bddd67613d0/volumes" Oct 31 00:41:33.485733 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.485712 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-fxb85"] Oct 31 00:41:33.490256 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.490241 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" Oct 31 00:41:33.494363 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.494324 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-collector-dockercfg-fkhp6\"" Oct 31 00:41:33.494363 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.494334 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"kube-root-ca.crt\"" Oct 31 00:41:33.494363 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.494323 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-collector-token\"" Oct 31 00:41:33.494363 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.494329 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-metrics\"" Oct 31 00:41:33.494575 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.494366 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"openshift-service-ca.crt\"" Oct 31 00:41:33.495866 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.495837 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-config\"" Oct 31 00:41:33.497678 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.497658 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-trustbundle\"" Oct 31 00:41:33.502326 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.502305 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-fxb85"] Oct 31 00:41:33.562242 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.562219 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-data-science-cluster-log-forwarder-collector-token\") pod \"data-science-cluster-log-forwarder-fxb85\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" Oct 31 00:41:33.562383 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.562248 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-sa-token\") pod \"data-science-cluster-log-forwarder-fxb85\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" Oct 31 00:41:33.562383 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.562271 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-metrics\") pod \"data-science-cluster-log-forwarder-fxb85\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" Oct 31 00:41:33.562383 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.562322 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-datadir\") pod \"data-science-cluster-log-forwarder-fxb85\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" Oct 31 00:41:33.562496 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.562452 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-config-openshift-service-cacrt\") pod \"data-science-cluster-log-forwarder-fxb85\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" Oct 31 00:41:33.562496 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.562470 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-config\") pod \"data-science-cluster-log-forwarder-fxb85\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" Oct 31 00:41:33.562496 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.562491 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xtxg\" (UniqueName: \"kubernetes.io/projected/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-kube-api-access-9xtxg\") pod \"data-science-cluster-log-forwarder-fxb85\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" Oct 31 00:41:33.562594 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.562510 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-trusted-ca\") pod \"data-science-cluster-log-forwarder-fxb85\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" Oct 31 00:41:33.562648 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.562625 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-tmp\") pod \"data-science-cluster-log-forwarder-fxb85\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" Oct 31 00:41:33.562725 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.562711 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-entrypoint\") pod \"data-science-cluster-log-forwarder-fxb85\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" Oct 31 00:41:33.562767 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.562756 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-varlogpods\") pod \"data-science-cluster-log-forwarder-fxb85\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" Oct 31 00:41:33.663287 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.663263 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-tmp\") pod \"data-science-cluster-log-forwarder-fxb85\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" Oct 31 00:41:33.663403 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.663306 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-entrypoint\") pod \"data-science-cluster-log-forwarder-fxb85\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" Oct 31 00:41:33.663475 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.663423 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-varlogpods\") pod \"data-science-cluster-log-forwarder-fxb85\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" Oct 31 00:41:33.663475 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.663453 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-data-science-cluster-log-forwarder-collector-token\") pod \"data-science-cluster-log-forwarder-fxb85\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" Oct 31 00:41:33.663590 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.663476 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-sa-token\") pod \"data-science-cluster-log-forwarder-fxb85\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" Oct 31 00:41:33.663590 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.663515 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-metrics\") pod \"data-science-cluster-log-forwarder-fxb85\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" Oct 31 00:41:33.663590 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.663518 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-varlogpods\") pod \"data-science-cluster-log-forwarder-fxb85\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" Oct 31 00:41:33.663590 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.663538 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-datadir\") pod \"data-science-cluster-log-forwarder-fxb85\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" Oct 31 00:41:33.663590 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.663582 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-config-openshift-service-cacrt\") pod \"data-science-cluster-log-forwarder-fxb85\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" Oct 31 00:41:33.663847 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.663605 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-config\") pod \"data-science-cluster-log-forwarder-fxb85\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" Oct 31 00:41:33.663847 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.663636 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xtxg\" (UniqueName: \"kubernetes.io/projected/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-kube-api-access-9xtxg\") pod \"data-science-cluster-log-forwarder-fxb85\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" Oct 31 00:41:33.663847 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.663633 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-datadir\") pod \"data-science-cluster-log-forwarder-fxb85\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" Oct 31 00:41:33.663847 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.663661 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-trusted-ca\") pod \"data-science-cluster-log-forwarder-fxb85\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" Oct 31 00:41:33.672823 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.664518 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-entrypoint\") pod \"data-science-cluster-log-forwarder-fxb85\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" Oct 31 00:41:33.672823 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.664755 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-trusted-ca\") pod \"data-science-cluster-log-forwarder-fxb85\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" Oct 31 00:41:33.672823 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.664942 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-config-openshift-service-cacrt\") pod \"data-science-cluster-log-forwarder-fxb85\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" Oct 31 00:41:33.672823 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.665424 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-config\") pod \"data-science-cluster-log-forwarder-fxb85\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" Oct 31 00:41:33.672823 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.665951 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-tmp\") pod \"data-science-cluster-log-forwarder-fxb85\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" Oct 31 00:41:33.672823 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.666693 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-metrics\") pod \"data-science-cluster-log-forwarder-fxb85\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" Oct 31 00:41:33.673237 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.673219 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-data-science-cluster-log-forwarder-collector-token\") pod \"data-science-cluster-log-forwarder-fxb85\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" Oct 31 00:41:33.675571 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.675544 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-sa-token\") pod \"data-science-cluster-log-forwarder-fxb85\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" Oct 31 00:41:33.675743 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.675725 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xtxg\" (UniqueName: \"kubernetes.io/projected/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-kube-api-access-9xtxg\") pod \"data-science-cluster-log-forwarder-fxb85\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" Oct 31 00:41:33.803420 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:33.803360 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" Oct 31 00:41:34.080522 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:34.080468 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-fxb85"] Oct 31 00:41:34.082870 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:41:34.082837 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03eba0ee_bc27_4c6a_9648_7fbeaec6e4b8.slice/crio-105a9010ca23901d07bdeb8af32d40dac82646e04ae9b7f8fe242638985dcfbb WatchSource:0}: Error finding container 105a9010ca23901d07bdeb8af32d40dac82646e04ae9b7f8fe242638985dcfbb: Status 404 returned error can't find the container with id 105a9010ca23901d07bdeb8af32d40dac82646e04ae9b7f8fe242638985dcfbb Oct 31 00:41:34.435530 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:34.435499 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" event={"ID":"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8","Type":"ContainerStarted","Data":"105a9010ca23901d07bdeb8af32d40dac82646e04ae9b7f8fe242638985dcfbb"} Oct 31 00:41:38.397266 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:38.397241 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5f5775896-fc8b7" Oct 31 00:41:38.459071 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:38.459028 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" event={"ID":"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8","Type":"ContainerStarted","Data":"9e4072085677fa0e22e33dace5b7f986afa15d096170aea3910082fe42115d82"} Oct 31 00:41:38.482552 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:41:38.482507 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" podStartSLOduration=1.647226385 podStartE2EDuration="5.482494094s" totalImagesPullingTime="3.835267709s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:41:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:38.481231313 +0000 UTC m=+2535.118050029" watchObservedRunningTime="2026-10-31 00:41:38.482494094 +0000 UTC m=+2535.119312809" Oct 31 00:41:50.899593 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:41:50.899551 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:42:20.898727 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:42:20.898689 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:42:45.342463 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.342425 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5f5775896-fc8b7"] Oct 31 00:42:45.342990 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.342844 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5f5775896-fc8b7" podUID="b80e91de-e9e3-4c8f-9791-65c350a2e101" containerName="korrel8r" containerID="cri-o://32dc6711886bec86075a576220e4d17287466df8f897a06ba8355b525acb4eca" gracePeriod=30 Oct 31 00:42:45.567331 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.567295 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb"] Oct 31 00:42:45.567528 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.567487 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" podUID="c2141e01-9d1a-4688-a9b3-cc501d0edbf9" containerName="gateway" containerID="cri-o://e5aa4c74d61285c4a418914d4fd343290dccb087eab52910db959b122993b38e" gracePeriod=30 Oct 31 00:42:45.567823 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.567798 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" podUID="c2141e01-9d1a-4688-a9b3-cc501d0edbf9" containerName="opa" containerID="cri-o://1b1998d006e70929ca79a121419451a4dc67905528257fde650b1146aafca7ef" gracePeriod=30 Oct 31 00:42:45.598090 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.598012 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-fxb85"] Oct 31 00:42:45.602285 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.601906 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" podUID="03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8" containerName="collector" containerID="cri-o://9e4072085677fa0e22e33dace5b7f986afa15d096170aea3910082fe42115d82" gracePeriod=10 Oct 31 00:42:45.718334 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.718311 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5f5775896-fc8b7" Oct 31 00:42:45.781052 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.781021 5274 generic.go:356] "Generic (PLEG): container finished" podID="c2141e01-9d1a-4688-a9b3-cc501d0edbf9" containerID="1b1998d006e70929ca79a121419451a4dc67905528257fde650b1146aafca7ef" exitCode=0 Oct 31 00:42:45.781176 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.781089 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" event={"ID":"c2141e01-9d1a-4688-a9b3-cc501d0edbf9","Type":"ContainerDied","Data":"1b1998d006e70929ca79a121419451a4dc67905528257fde650b1146aafca7ef"} Oct 31 00:42:45.782476 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.782454 5274 generic.go:356] "Generic (PLEG): container finished" podID="b80e91de-e9e3-4c8f-9791-65c350a2e101" containerID="32dc6711886bec86075a576220e4d17287466df8f897a06ba8355b525acb4eca" exitCode=0 Oct 31 00:42:45.782585 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.782507 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5f5775896-fc8b7" Oct 31 00:42:45.782585 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.782534 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5f5775896-fc8b7" event={"ID":"b80e91de-e9e3-4c8f-9791-65c350a2e101","Type":"ContainerDied","Data":"32dc6711886bec86075a576220e4d17287466df8f897a06ba8355b525acb4eca"} Oct 31 00:42:45.782585 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.782583 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5f5775896-fc8b7" event={"ID":"b80e91de-e9e3-4c8f-9791-65c350a2e101","Type":"ContainerDied","Data":"9e1453efcbe727a104c6de4e6724d44cb8e2e39609a715b55692a99212b92d7d"} Oct 31 00:42:45.782744 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.782604 5274 scope.go:122] "RemoveContainer" containerID="32dc6711886bec86075a576220e4d17287466df8f897a06ba8355b525acb4eca" Oct 31 00:42:45.784489 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.784468 5274 generic.go:356] "Generic (PLEG): container finished" podID="03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8" containerID="9e4072085677fa0e22e33dace5b7f986afa15d096170aea3910082fe42115d82" exitCode=0 Oct 31 00:42:45.784598 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.784537 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" event={"ID":"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8","Type":"ContainerDied","Data":"9e4072085677fa0e22e33dace5b7f986afa15d096170aea3910082fe42115d82"} Oct 31 00:42:45.792334 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.792311 5274 scope.go:122] "RemoveContainer" containerID="32dc6711886bec86075a576220e4d17287466df8f897a06ba8355b525acb4eca" Oct 31 00:42:45.792643 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:42:45.792617 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32dc6711886bec86075a576220e4d17287466df8f897a06ba8355b525acb4eca\": container with ID starting with 32dc6711886bec86075a576220e4d17287466df8f897a06ba8355b525acb4eca not found: ID does not exist" containerID="32dc6711886bec86075a576220e4d17287466df8f897a06ba8355b525acb4eca" Oct 31 00:42:45.792737 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.792655 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32dc6711886bec86075a576220e4d17287466df8f897a06ba8355b525acb4eca"} err="failed to get container status \"32dc6711886bec86075a576220e4d17287466df8f897a06ba8355b525acb4eca\": rpc error: code = NotFound desc = could not find container \"32dc6711886bec86075a576220e4d17287466df8f897a06ba8355b525acb4eca\": container with ID starting with 32dc6711886bec86075a576220e4d17287466df8f897a06ba8355b525acb4eca not found: ID does not exist" Oct 31 00:42:45.838476 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.838454 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b80e91de-e9e3-4c8f-9791-65c350a2e101-config\" (UniqueName: \"kubernetes.io/configmap/b80e91de-e9e3-4c8f-9791-65c350a2e101-config\") pod \"b80e91de-e9e3-4c8f-9791-65c350a2e101\" (UID: \"b80e91de-e9e3-4c8f-9791-65c350a2e101\") " Oct 31 00:42:45.838588 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.838489 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b80e91de-e9e3-4c8f-9791-65c350a2e101-kube-api-access-x79ll\" (UniqueName: \"kubernetes.io/projected/b80e91de-e9e3-4c8f-9791-65c350a2e101-kube-api-access-x79ll\") pod \"b80e91de-e9e3-4c8f-9791-65c350a2e101\" (UID: \"b80e91de-e9e3-4c8f-9791-65c350a2e101\") " Oct 31 00:42:45.838631 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.838596 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b80e91de-e9e3-4c8f-9791-65c350a2e101-tls\" (UniqueName: \"kubernetes.io/secret/b80e91de-e9e3-4c8f-9791-65c350a2e101-tls\") pod \"b80e91de-e9e3-4c8f-9791-65c350a2e101\" (UID: \"b80e91de-e9e3-4c8f-9791-65c350a2e101\") " Oct 31 00:42:45.838836 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.838812 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b80e91de-e9e3-4c8f-9791-65c350a2e101-config" pod "b80e91de-e9e3-4c8f-9791-65c350a2e101" (UID: "b80e91de-e9e3-4c8f-9791-65c350a2e101"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:45.840605 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.840577 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b80e91de-e9e3-4c8f-9791-65c350a2e101-kube-api-access-x79ll" pod "b80e91de-e9e3-4c8f-9791-65c350a2e101" (UID: "b80e91de-e9e3-4c8f-9791-65c350a2e101"). InnerVolumeSpecName "kube-api-access-x79ll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:45.840985 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.840965 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b80e91de-e9e3-4c8f-9791-65c350a2e101-tls" pod "b80e91de-e9e3-4c8f-9791-65c350a2e101" (UID: "b80e91de-e9e3-4c8f-9791-65c350a2e101"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:45.850100 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.850053 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" Oct 31 00:42:45.939574 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.939548 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-config\" (UniqueName: \"kubernetes.io/configmap/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-config\") pod \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " Oct 31 00:42:45.939705 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.939580 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/host-path/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-varlogpods\" (UniqueName: \"kubernetes.io/host-path/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-varlogpods\") pod \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " Oct 31 00:42:45.939705 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.939614 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-tmp\" (UniqueName: \"kubernetes.io/empty-dir/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-tmp\") pod \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " Oct 31 00:42:45.939705 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.939645 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-sa-token\" (UniqueName: \"kubernetes.io/projected/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-sa-token\") pod \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " Oct 31 00:42:45.939705 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.939652 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-varlogpods" pod "03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8" (UID: "03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8"). InnerVolumeSpecName "varlogpods". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Oct 31 00:42:45.939705 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.939672 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-trusted-ca\") pod \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " Oct 31 00:42:45.940008 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.939710 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-metrics\" (UniqueName: \"kubernetes.io/secret/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-metrics\") pod \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " Oct 31 00:42:45.940008 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.939737 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-entrypoint\" (UniqueName: \"kubernetes.io/configmap/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-entrypoint\") pod \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " Oct 31 00:42:45.940008 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.939789 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/host-path/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-datadir\" (UniqueName: \"kubernetes.io/host-path/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-datadir\") pod \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " Oct 31 00:42:45.940008 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.939827 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-config-openshift-service-cacrt\") pod \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " Oct 31 00:42:45.940008 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.939860 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-data-science-cluster-log-forwarder-collector-token\") pod \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " Oct 31 00:42:45.940008 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.939905 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-kube-api-access-9xtxg\" (UniqueName: \"kubernetes.io/projected/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-kube-api-access-9xtxg\") pod \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\" (UID: \"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8\") " Oct 31 00:42:45.940008 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.939921 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-config" pod "03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8" (UID: "03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:45.940380 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.940039 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-trusted-ca" pod "03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8" (UID: "03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:45.940380 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.940287 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-datadir" pod "03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8" (UID: "03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8"). InnerVolumeSpecName "datadir". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Oct 31 00:42:45.940380 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.940367 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:45.940535 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.940388 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b80e91de-e9e3-4c8f-9791-65c350a2e101-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:45.940535 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.940401 5274 reconciler_common.go:299] "Volume detached for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-varlogpods\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:45.940535 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.940417 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b80e91de-e9e3-4c8f-9791-65c350a2e101-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:45.940535 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.940432 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x79ll\" (UniqueName: \"kubernetes.io/projected/b80e91de-e9e3-4c8f-9791-65c350a2e101-kube-api-access-x79ll\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:45.940535 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.940446 5274 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-trusted-ca\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:45.940535 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.940366 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-config-openshift-service-cacrt" pod "03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8" (UID: "03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8"). InnerVolumeSpecName "config-openshift-service-cacrt". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:45.940535 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.940460 5274 reconciler_common.go:299] "Volume detached for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-datadir\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:45.942307 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.942279 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-sa-token" pod "03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8" (UID: "03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8"). InnerVolumeSpecName "sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:45.942848 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.942812 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-kube-api-access-9xtxg" pod "03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8" (UID: "03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8"). InnerVolumeSpecName "kube-api-access-9xtxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:45.942975 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.942883 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-metrics" pod "03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8" (UID: "03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8"). InnerVolumeSpecName "metrics". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:45.943073 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.943048 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-data-science-cluster-log-forwarder-collector-token" pod "03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8" (UID: "03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8"). InnerVolumeSpecName "data-science-cluster-log-forwarder-collector-token". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:45.943209 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.943098 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-tmp" pod "03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8" (UID: "03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:45.945581 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:45.945559 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-entrypoint" pod "03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8" (UID: "03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8"). InnerVolumeSpecName "entrypoint". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:46.041271 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.041245 5274 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-tmp\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.041271 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.041265 5274 reconciler_common.go:299] "Volume detached for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-sa-token\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.041271 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.041273 5274 reconciler_common.go:299] "Volume detached for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-metrics\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.041473 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.041281 5274 reconciler_common.go:299] "Volume detached for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-entrypoint\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.041473 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.041291 5274 reconciler_common.go:299] "Volume detached for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-config-openshift-service-cacrt\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.041473 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.041301 5274 reconciler_common.go:299] "Volume detached for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-data-science-cluster-log-forwarder-collector-token\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.041473 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.041310 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xtxg\" (UniqueName: \"kubernetes.io/projected/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8-kube-api-access-9xtxg\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.108680 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.108626 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5f5775896-fc8b7"] Oct 31 00:42:46.113002 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.112979 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5f5775896-fc8b7"] Oct 31 00:42:46.745538 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.745507 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-ktqtv"] Oct 31 00:42:46.745974 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.745902 5274 cpu_manager.go:414] "containerMap: removing container" podUID="b80e91de-e9e3-4c8f-9791-65c350a2e101" containerName="korrel8r" Oct 31 00:42:46.745974 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.745919 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b80e91de-e9e3-4c8f-9791-65c350a2e101" containerName="korrel8r" Oct 31 00:42:46.745974 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.745942 5274 cpu_manager.go:414] "containerMap: removing container" podUID="03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8" containerName="collector" Oct 31 00:42:46.745974 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.745971 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8" containerName="collector" Oct 31 00:42:46.746233 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.746082 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="b80e91de-e9e3-4c8f-9791-65c350a2e101" containerName="korrel8r" Oct 31 00:42:46.746233 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.746098 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8" containerName="collector" Oct 31 00:42:46.749243 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.749221 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-ktqtv" Oct 31 00:42:46.763943 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.763921 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-ktqtv"] Oct 31 00:42:46.792188 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.792158 5274 generic.go:356] "Generic (PLEG): container finished" podID="c2141e01-9d1a-4688-a9b3-cc501d0edbf9" containerID="e5aa4c74d61285c4a418914d4fd343290dccb087eab52910db959b122993b38e" exitCode=0 Oct 31 00:42:46.792339 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.792229 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" event={"ID":"c2141e01-9d1a-4688-a9b3-cc501d0edbf9","Type":"ContainerDied","Data":"e5aa4c74d61285c4a418914d4fd343290dccb087eab52910db959b122993b38e"} Oct 31 00:42:46.794351 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.794328 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" event={"ID":"03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8","Type":"ContainerDied","Data":"105a9010ca23901d07bdeb8af32d40dac82646e04ae9b7f8fe242638985dcfbb"} Oct 31 00:42:46.794479 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.794361 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-fxb85" Oct 31 00:42:46.794479 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.794379 5274 scope.go:122] "RemoveContainer" containerID="9e4072085677fa0e22e33dace5b7f986afa15d096170aea3910082fe42115d82" Oct 31 00:42:46.819619 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.819597 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-fxb85"] Oct 31 00:42:46.825757 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.825735 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-fxb85"] Oct 31 00:42:46.831562 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.831544 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" Oct 31 00:42:46.848434 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.848416 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6fqz\" (UniqueName: \"kubernetes.io/projected/e32e8e22-a821-4b4a-9f93-1983ebb326c0-kube-api-access-v6fqz\") pod \"data-science-usage-logs-collector-9f68bf844-ktqtv\" (UID: \"e32e8e22-a821-4b4a-9f93-1983ebb326c0\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-ktqtv" Oct 31 00:42:46.848516 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.848453 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e32e8e22-a821-4b4a-9f93-1983ebb326c0-otc-internal\") pod \"data-science-usage-logs-collector-9f68bf844-ktqtv\" (UID: \"e32e8e22-a821-4b4a-9f93-1983ebb326c0\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-ktqtv" Oct 31 00:42:46.848577 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.848560 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e32e8e22-a821-4b4a-9f93-1983ebb326c0-otlp-tls-certs\") pod \"data-science-usage-logs-collector-9f68bf844-ktqtv\" (UID: \"e32e8e22-a821-4b4a-9f93-1983ebb326c0\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-ktqtv" Oct 31 00:42:46.949512 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.949456 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-kube-api-access-wxcn7\" (UniqueName: \"kubernetes.io/projected/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-kube-api-access-wxcn7\") pod \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\" (UID: \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\") " Oct 31 00:42:46.949512 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.949504 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-lokistack-gateway\") pod \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\" (UID: \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\") " Oct 31 00:42:46.949661 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.949533 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-tls-secret\" (UniqueName: \"kubernetes.io/projected/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-tls-secret\") pod \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\" (UID: \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\") " Oct 31 00:42:46.949661 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.949566 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-data-science-lokistack-ca-bundle\") pod \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\" (UID: \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\") " Oct 31 00:42:46.949661 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.949592 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-data-science-lokistack-gateway-ca-bundle\") pod \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\" (UID: \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\") " Oct 31 00:42:46.949661 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.949624 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-tenants\" (UniqueName: \"kubernetes.io/secret/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-tenants\") pod \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\" (UID: \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\") " Oct 31 00:42:46.949874 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.949669 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-rbac\" (UniqueName: \"kubernetes.io/configmap/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-rbac\") pod \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\" (UID: \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\") " Oct 31 00:42:46.949874 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.949709 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-data-science-lokistack-gateway-client-http\") pod \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\" (UID: \"c2141e01-9d1a-4688-a9b3-cc501d0edbf9\") " Oct 31 00:42:46.949874 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.949851 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e32e8e22-a821-4b4a-9f93-1983ebb326c0-otlp-tls-certs\") pod \"data-science-usage-logs-collector-9f68bf844-ktqtv\" (UID: \"e32e8e22-a821-4b4a-9f93-1983ebb326c0\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-ktqtv" Oct 31 00:42:46.950096 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.949920 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6fqz\" (UniqueName: \"kubernetes.io/projected/e32e8e22-a821-4b4a-9f93-1983ebb326c0-kube-api-access-v6fqz\") pod \"data-science-usage-logs-collector-9f68bf844-ktqtv\" (UID: \"e32e8e22-a821-4b4a-9f93-1983ebb326c0\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-ktqtv" Oct 31 00:42:46.950096 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.949964 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-data-science-lokistack-ca-bundle" pod "c2141e01-9d1a-4688-a9b3-cc501d0edbf9" (UID: "c2141e01-9d1a-4688-a9b3-cc501d0edbf9"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:46.950096 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.950067 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e32e8e22-a821-4b4a-9f93-1983ebb326c0-otc-internal\") pod \"data-science-usage-logs-collector-9f68bf844-ktqtv\" (UID: \"e32e8e22-a821-4b4a-9f93-1983ebb326c0\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-ktqtv" Oct 31 00:42:46.950259 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.950179 5274 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-data-science-lokistack-ca-bundle\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.951050 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.951024 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e32e8e22-a821-4b4a-9f93-1983ebb326c0-otc-internal\") pod \"data-science-usage-logs-collector-9f68bf844-ktqtv\" (UID: \"e32e8e22-a821-4b4a-9f93-1983ebb326c0\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-ktqtv" Oct 31 00:42:46.952350 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.952323 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-kube-api-access-wxcn7" pod "c2141e01-9d1a-4688-a9b3-cc501d0edbf9" (UID: "c2141e01-9d1a-4688-a9b3-cc501d0edbf9"). InnerVolumeSpecName "kube-api-access-wxcn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:46.952350 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.952337 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-tls-secret" pod "c2141e01-9d1a-4688-a9b3-cc501d0edbf9" (UID: "c2141e01-9d1a-4688-a9b3-cc501d0edbf9"). InnerVolumeSpecName "tls-secret". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:46.952468 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.952418 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-data-science-lokistack-gateway-ca-bundle" pod "c2141e01-9d1a-4688-a9b3-cc501d0edbf9" (UID: "c2141e01-9d1a-4688-a9b3-cc501d0edbf9"). InnerVolumeSpecName "data-science-lokistack-gateway-ca-bundle". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:46.952532 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.952507 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-data-science-lokistack-gateway-client-http" pod "c2141e01-9d1a-4688-a9b3-cc501d0edbf9" (UID: "c2141e01-9d1a-4688-a9b3-cc501d0edbf9"). InnerVolumeSpecName "data-science-lokistack-gateway-client-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:46.953055 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.953036 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e32e8e22-a821-4b4a-9f93-1983ebb326c0-otlp-tls-certs\") pod \"data-science-usage-logs-collector-9f68bf844-ktqtv\" (UID: \"e32e8e22-a821-4b4a-9f93-1983ebb326c0\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-ktqtv" Oct 31 00:42:46.956484 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.956459 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-lokistack-gateway" pod "c2141e01-9d1a-4688-a9b3-cc501d0edbf9" (UID: "c2141e01-9d1a-4688-a9b3-cc501d0edbf9"). InnerVolumeSpecName "lokistack-gateway". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:46.956484 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.956468 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-tenants" pod "c2141e01-9d1a-4688-a9b3-cc501d0edbf9" (UID: "c2141e01-9d1a-4688-a9b3-cc501d0edbf9"). InnerVolumeSpecName "tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:46.956855 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.956838 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-rbac" pod "c2141e01-9d1a-4688-a9b3-cc501d0edbf9" (UID: "c2141e01-9d1a-4688-a9b3-cc501d0edbf9"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:46.962006 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:46.961968 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6fqz\" (UniqueName: \"kubernetes.io/projected/e32e8e22-a821-4b4a-9f93-1983ebb326c0-kube-api-access-v6fqz\") pod \"data-science-usage-logs-collector-9f68bf844-ktqtv\" (UID: \"e32e8e22-a821-4b4a-9f93-1983ebb326c0\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-ktqtv" Oct 31 00:42:47.051082 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:47.051052 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxcn7\" (UniqueName: \"kubernetes.io/projected/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-kube-api-access-wxcn7\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:47.051082 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:47.051076 5274 reconciler_common.go:299] "Volume detached for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-lokistack-gateway\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:47.051082 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:47.051087 5274 reconciler_common.go:299] "Volume detached for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-tls-secret\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:47.051298 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:47.051105 5274 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-data-science-lokistack-gateway-ca-bundle\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:47.051298 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:47.051115 5274 reconciler_common.go:299] "Volume detached for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-tenants\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:47.051298 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:47.051123 5274 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-rbac\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:47.051298 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:47.051131 5274 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/c2141e01-9d1a-4688-a9b3-cc501d0edbf9-data-science-lokistack-gateway-client-http\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:47.066255 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:47.066236 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-ktqtv" Oct 31 00:42:47.348847 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:47.348796 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-ktqtv"] Oct 31 00:42:47.351128 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:42:47.351092 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode32e8e22_a821_4b4a_9f93_1983ebb326c0.slice/crio-174bad2b2858ca821ef5ff9c48d462e665224164bb976698b0ffa791fb95ad2e WatchSource:0}: Error finding container 174bad2b2858ca821ef5ff9c48d462e665224164bb976698b0ffa791fb95ad2e: Status 404 returned error can't find the container with id 174bad2b2858ca821ef5ff9c48d462e665224164bb976698b0ffa791fb95ad2e Oct 31 00:42:47.353249 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:47.353218 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:42:47.353334 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:47.353276 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:42:47.481605 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:47.481579 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8" path="/var/lib/kubelet/pods/03eba0ee-bc27-4c6a-9648-7fbeaec6e4b8/volumes" Oct 31 00:42:47.482002 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:47.481966 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b80e91de-e9e3-4c8f-9791-65c350a2e101" path="/var/lib/kubelet/pods/b80e91de-e9e3-4c8f-9791-65c350a2e101/volumes" Oct 31 00:42:47.801752 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:47.801716 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-ktqtv" event={"ID":"e32e8e22-a821-4b4a-9f93-1983ebb326c0","Type":"ContainerStarted","Data":"c648636f28bc746b12d62c8121756b4094c0c12e5ea4068b8162117ad68b7ff2"} Oct 31 00:42:47.802284 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:47.801779 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-ktqtv" event={"ID":"e32e8e22-a821-4b4a-9f93-1983ebb326c0","Type":"ContainerStarted","Data":"174bad2b2858ca821ef5ff9c48d462e665224164bb976698b0ffa791fb95ad2e"} Oct 31 00:42:47.803471 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:47.803450 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" Oct 31 00:42:47.803471 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:47.803461 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb" event={"ID":"c2141e01-9d1a-4688-a9b3-cc501d0edbf9","Type":"ContainerDied","Data":"3c278f354b06a05c9180470b1e1cc94d19ce1dfa2f9006a6cd377696814ec2c9"} Oct 31 00:42:47.803639 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:47.803498 5274 scope.go:122] "RemoveContainer" containerID="1b1998d006e70929ca79a121419451a4dc67905528257fde650b1146aafca7ef" Oct 31 00:42:47.811813 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:47.811797 5274 scope.go:122] "RemoveContainer" containerID="e5aa4c74d61285c4a418914d4fd343290dccb087eab52910db959b122993b38e" Oct 31 00:42:47.821416 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:47.821376 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-ktqtv" podStartSLOduration=1.8213643990000001 podStartE2EDuration="1.821364399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:47.821222235 +0000 UTC m=+2604.458040951" watchObservedRunningTime="2026-10-31 00:42:47.821364399 +0000 UTC m=+2604.458183151" Oct 31 00:42:47.838239 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:47.838215 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb"] Oct 31 00:42:47.845308 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:47.845288 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-66cfb97849-fd4tb"] Oct 31 00:42:47.851919 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:47.851899 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-74s62"] Oct 31 00:42:47.852243 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:47.852179 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-74s62" podUID="95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f" containerName="otc-container" containerID="cri-o://b2874018387d363f0e326e2b804c7caccb6ac005d15eeceabd8740115209f039" gracePeriod=30 Oct 31 00:42:48.096423 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:48.096403 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-74s62" Oct 31 00:42:48.263096 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:48.263065 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f-otc-internal\" (UniqueName: \"kubernetes.io/configmap/95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f-otc-internal\") pod \"95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f\" (UID: \"95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f\") " Oct 31 00:42:48.263269 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:48.263152 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f-kube-api-access-p6zsj\" (UniqueName: \"kubernetes.io/projected/95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f-kube-api-access-p6zsj\") pod \"95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f\" (UID: \"95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f\") " Oct 31 00:42:48.263269 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:48.263187 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f-otlp-tls-certs\") pod \"95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f\" (UID: \"95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f\") " Oct 31 00:42:48.263442 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:48.263420 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f-otc-internal" pod "95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f" (UID: "95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:48.265233 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:48.265210 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f-kube-api-access-p6zsj" pod "95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f" (UID: "95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f"). InnerVolumeSpecName "kube-api-access-p6zsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:48.265311 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:48.265233 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f-otlp-tls-certs" pod "95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f" (UID: "95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:48.364332 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:48.364284 5274 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f-otc-internal\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:48.364332 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:48.364304 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6zsj\" (UniqueName: \"kubernetes.io/projected/95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f-kube-api-access-p6zsj\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:48.364332 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:48.364313 5274 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f-otlp-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:48.811345 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:48.811315 5274 generic.go:356] "Generic (PLEG): container finished" podID="95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f" containerID="b2874018387d363f0e326e2b804c7caccb6ac005d15eeceabd8740115209f039" exitCode=0 Oct 31 00:42:48.811783 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:48.811362 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-74s62" Oct 31 00:42:48.811783 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:48.811400 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-74s62" event={"ID":"95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f","Type":"ContainerDied","Data":"b2874018387d363f0e326e2b804c7caccb6ac005d15eeceabd8740115209f039"} Oct 31 00:42:48.811783 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:48.811451 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-74s62" event={"ID":"95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f","Type":"ContainerDied","Data":"bd71fae510f0ba6841d8453ed2ec5a424f166243bdbc0c2190c22551ee359d46"} Oct 31 00:42:48.811783 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:48.811477 5274 scope.go:122] "RemoveContainer" containerID="b2874018387d363f0e326e2b804c7caccb6ac005d15eeceabd8740115209f039" Oct 31 00:42:48.820627 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:48.820612 5274 scope.go:122] "RemoveContainer" containerID="b2874018387d363f0e326e2b804c7caccb6ac005d15eeceabd8740115209f039" Oct 31 00:42:48.820886 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:42:48.820867 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2874018387d363f0e326e2b804c7caccb6ac005d15eeceabd8740115209f039\": container with ID starting with b2874018387d363f0e326e2b804c7caccb6ac005d15eeceabd8740115209f039 not found: ID does not exist" containerID="b2874018387d363f0e326e2b804c7caccb6ac005d15eeceabd8740115209f039" Oct 31 00:42:48.820926 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:48.820896 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2874018387d363f0e326e2b804c7caccb6ac005d15eeceabd8740115209f039"} err="failed to get container status \"b2874018387d363f0e326e2b804c7caccb6ac005d15eeceabd8740115209f039\": rpc error: code = NotFound desc = could not find container \"b2874018387d363f0e326e2b804c7caccb6ac005d15eeceabd8740115209f039\": container with ID starting with b2874018387d363f0e326e2b804c7caccb6ac005d15eeceabd8740115209f039 not found: ID does not exist" Oct 31 00:42:48.833800 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:48.833778 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-74s62"] Oct 31 00:42:48.838484 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:48.838465 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-74s62"] Oct 31 00:42:49.481653 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:49.481624 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f" path="/var/lib/kubelet/pods/95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f/volumes" Oct 31 00:42:49.482031 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:49.482018 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2141e01-9d1a-4688-a9b3-cc501d0edbf9" path="/var/lib/kubelet/pods/c2141e01-9d1a-4688-a9b3-cc501d0edbf9/volumes" Oct 31 00:42:50.898282 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:42:50.898246 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:42:53.091568 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:53.091539 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-6mgrd"] Oct 31 00:42:53.091936 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:53.091842 5274 cpu_manager.go:414] "containerMap: removing container" podUID="95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f" containerName="otc-container" Oct 31 00:42:53.091936 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:53.091852 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f" containerName="otc-container" Oct 31 00:42:53.091936 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:53.091880 5274 cpu_manager.go:414] "containerMap: removing container" podUID="c2141e01-9d1a-4688-a9b3-cc501d0edbf9" containerName="opa" Oct 31 00:42:53.091936 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:53.091886 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2141e01-9d1a-4688-a9b3-cc501d0edbf9" containerName="opa" Oct 31 00:42:53.091936 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:53.091896 5274 cpu_manager.go:414] "containerMap: removing container" podUID="c2141e01-9d1a-4688-a9b3-cc501d0edbf9" containerName="gateway" Oct 31 00:42:53.091936 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:53.091901 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2141e01-9d1a-4688-a9b3-cc501d0edbf9" containerName="gateway" Oct 31 00:42:53.092147 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:53.091970 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2141e01-9d1a-4688-a9b3-cc501d0edbf9" containerName="opa" Oct 31 00:42:53.092147 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:53.091981 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2141e01-9d1a-4688-a9b3-cc501d0edbf9" containerName="gateway" Oct 31 00:42:53.092147 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:53.091990 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="95a57f36-ed6e-4923-a6ac-dd47ea0d5d5f" containerName="otc-container" Oct 31 00:42:53.095938 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:53.095917 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-6mgrd" Oct 31 00:42:53.106107 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:53.106085 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-6mgrd"] Oct 31 00:42:53.202644 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:53.202616 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/98049138-d8ad-41fc-9dc8-8347eb76f6c2-otlp-tls-certs\") pod \"data-science-usage-logs-collector-5b5b9f5c4f-6mgrd\" (UID: \"98049138-d8ad-41fc-9dc8-8347eb76f6c2\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-6mgrd" Oct 31 00:42:53.202776 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:53.202668 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sbgl\" (UniqueName: \"kubernetes.io/projected/98049138-d8ad-41fc-9dc8-8347eb76f6c2-kube-api-access-5sbgl\") pod \"data-science-usage-logs-collector-5b5b9f5c4f-6mgrd\" (UID: \"98049138-d8ad-41fc-9dc8-8347eb76f6c2\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-6mgrd" Oct 31 00:42:53.202776 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:53.202719 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/98049138-d8ad-41fc-9dc8-8347eb76f6c2-otc-internal\") pod \"data-science-usage-logs-collector-5b5b9f5c4f-6mgrd\" (UID: \"98049138-d8ad-41fc-9dc8-8347eb76f6c2\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-6mgrd" Oct 31 00:42:53.303117 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:53.303087 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sbgl\" (UniqueName: \"kubernetes.io/projected/98049138-d8ad-41fc-9dc8-8347eb76f6c2-kube-api-access-5sbgl\") pod \"data-science-usage-logs-collector-5b5b9f5c4f-6mgrd\" (UID: \"98049138-d8ad-41fc-9dc8-8347eb76f6c2\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-6mgrd" Oct 31 00:42:53.303117 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:53.303120 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/98049138-d8ad-41fc-9dc8-8347eb76f6c2-otc-internal\") pod \"data-science-usage-logs-collector-5b5b9f5c4f-6mgrd\" (UID: \"98049138-d8ad-41fc-9dc8-8347eb76f6c2\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-6mgrd" Oct 31 00:42:53.303315 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:53.303174 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/98049138-d8ad-41fc-9dc8-8347eb76f6c2-otlp-tls-certs\") pod \"data-science-usage-logs-collector-5b5b9f5c4f-6mgrd\" (UID: \"98049138-d8ad-41fc-9dc8-8347eb76f6c2\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-6mgrd" Oct 31 00:42:53.303864 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:53.303844 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/98049138-d8ad-41fc-9dc8-8347eb76f6c2-otc-internal\") pod \"data-science-usage-logs-collector-5b5b9f5c4f-6mgrd\" (UID: \"98049138-d8ad-41fc-9dc8-8347eb76f6c2\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-6mgrd" Oct 31 00:42:53.305622 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:53.305600 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/98049138-d8ad-41fc-9dc8-8347eb76f6c2-otlp-tls-certs\") pod \"data-science-usage-logs-collector-5b5b9f5c4f-6mgrd\" (UID: \"98049138-d8ad-41fc-9dc8-8347eb76f6c2\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-6mgrd" Oct 31 00:42:53.315504 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:53.315479 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sbgl\" (UniqueName: \"kubernetes.io/projected/98049138-d8ad-41fc-9dc8-8347eb76f6c2-kube-api-access-5sbgl\") pod \"data-science-usage-logs-collector-5b5b9f5c4f-6mgrd\" (UID: \"98049138-d8ad-41fc-9dc8-8347eb76f6c2\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-6mgrd" Oct 31 00:42:53.410533 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:53.410508 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-6mgrd" Oct 31 00:42:53.681808 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:53.681787 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-6mgrd"] Oct 31 00:42:53.684424 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:42:53.684396 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98049138_d8ad_41fc_9dc8_8347eb76f6c2.slice/crio-8084974c1344b07307a04125b56c0609c4defded7014b4a5c967fcc92e9d4cec WatchSource:0}: Error finding container 8084974c1344b07307a04125b56c0609c4defded7014b4a5c967fcc92e9d4cec: Status 404 returned error can't find the container with id 8084974c1344b07307a04125b56c0609c4defded7014b4a5c967fcc92e9d4cec Oct 31 00:42:53.686524 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:53.686484 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:42:53.686602 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:53.686561 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:42:53.823400 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:53.823368 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-6mgrd"] Oct 31 00:42:53.829195 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:53.829166 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-ktqtv"] Oct 31 00:42:53.829393 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:53.829367 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-ktqtv" podUID="e32e8e22-a821-4b4a-9f93-1983ebb326c0" containerName="otc-container" containerID="cri-o://c648636f28bc746b12d62c8121756b4094c0c12e5ea4068b8162117ad68b7ff2" gracePeriod=30 Oct 31 00:42:53.849120 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:53.849089 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-6mgrd" event={"ID":"98049138-d8ad-41fc-9dc8-8347eb76f6c2","Type":"ContainerStarted","Data":"d240f5593abeca73523c81a2f4ab0ae8fcc12a83f2ad0a865207f5e3d059fbb6"} Oct 31 00:42:53.849252 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:53.849152 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-6mgrd" event={"ID":"98049138-d8ad-41fc-9dc8-8347eb76f6c2","Type":"ContainerStarted","Data":"8084974c1344b07307a04125b56c0609c4defded7014b4a5c967fcc92e9d4cec"} Oct 31 00:42:53.849372 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:53.849350 5274 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-6mgrd" secret="" err="secret \"data-science-usage-logs-collector-dockercfg-sm6jl\" not found" Oct 31 00:42:53.868893 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:53.868850 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-6mgrd" podStartSLOduration=0.868836726 podStartE2EDuration="868.836726ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:53.867798875 +0000 UTC m=+2610.504617591" watchObservedRunningTime="2026-10-31 00:42:53.868836726 +0000 UTC m=+2610.505655441" Oct 31 00:42:54.010863 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:42:54.010823 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-usage-logs-collector-headless-tls: secret "data-science-usage-logs-collector-headless-tls" not found Oct 31 00:42:54.011041 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:42:54.010900 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98049138-d8ad-41fc-9dc8-8347eb76f6c2-otlp-tls-certs podName:98049138-d8ad-41fc-9dc8-8347eb76f6c2 nodeName:}" failed. No retries permitted until 2026-10-31 00:42:54.510879371 +0000 UTC m=+2611.147698082 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/98049138-d8ad-41fc-9dc8-8347eb76f6c2-otlp-tls-certs") pod "data-science-usage-logs-collector-5b5b9f5c4f-6mgrd" (UID: "98049138-d8ad-41fc-9dc8-8347eb76f6c2") : secret "data-science-usage-logs-collector-headless-tls" not found Oct 31 00:42:54.011041 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:42:54.010832 5274 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-usage-logs-collector-9d8884b6: configmap "data-science-usage-logs-collector-9d8884b6" not found Oct 31 00:42:54.011196 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:42:54.011063 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/98049138-d8ad-41fc-9dc8-8347eb76f6c2-otc-internal podName:98049138-d8ad-41fc-9dc8-8347eb76f6c2 nodeName:}" failed. No retries permitted until 2026-10-31 00:42:54.511047749 +0000 UTC m=+2611.147866446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otc-internal" (UniqueName: "kubernetes.io/configmap/98049138-d8ad-41fc-9dc8-8347eb76f6c2-otc-internal") pod "data-science-usage-logs-collector-5b5b9f5c4f-6mgrd" (UID: "98049138-d8ad-41fc-9dc8-8347eb76f6c2") : configmap "data-science-usage-logs-collector-9d8884b6" not found Oct 31 00:42:54.066105 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:54.066080 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-ktqtv" Oct 31 00:42:54.212299 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:54.212229 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e32e8e22-a821-4b4a-9f93-1983ebb326c0-otc-internal\" (UniqueName: \"kubernetes.io/configmap/e32e8e22-a821-4b4a-9f93-1983ebb326c0-otc-internal\") pod \"e32e8e22-a821-4b4a-9f93-1983ebb326c0\" (UID: \"e32e8e22-a821-4b4a-9f93-1983ebb326c0\") " Oct 31 00:42:54.212299 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:54.212267 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e32e8e22-a821-4b4a-9f93-1983ebb326c0-kube-api-access-v6fqz\" (UniqueName: \"kubernetes.io/projected/e32e8e22-a821-4b4a-9f93-1983ebb326c0-kube-api-access-v6fqz\") pod \"e32e8e22-a821-4b4a-9f93-1983ebb326c0\" (UID: \"e32e8e22-a821-4b4a-9f93-1983ebb326c0\") " Oct 31 00:42:54.212299 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:54.212295 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e32e8e22-a821-4b4a-9f93-1983ebb326c0-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e32e8e22-a821-4b4a-9f93-1983ebb326c0-otlp-tls-certs\") pod \"e32e8e22-a821-4b4a-9f93-1983ebb326c0\" (UID: \"e32e8e22-a821-4b4a-9f93-1983ebb326c0\") " Oct 31 00:42:54.212733 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:54.212559 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e32e8e22-a821-4b4a-9f93-1983ebb326c0-otc-internal" pod "e32e8e22-a821-4b4a-9f93-1983ebb326c0" (UID: "e32e8e22-a821-4b4a-9f93-1983ebb326c0"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:54.214266 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:54.214246 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e32e8e22-a821-4b4a-9f93-1983ebb326c0-otlp-tls-certs" pod "e32e8e22-a821-4b4a-9f93-1983ebb326c0" (UID: "e32e8e22-a821-4b4a-9f93-1983ebb326c0"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:54.214323 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:54.214285 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e32e8e22-a821-4b4a-9f93-1983ebb326c0-kube-api-access-v6fqz" pod "e32e8e22-a821-4b4a-9f93-1983ebb326c0" (UID: "e32e8e22-a821-4b4a-9f93-1983ebb326c0"). InnerVolumeSpecName "kube-api-access-v6fqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:54.313269 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:54.313240 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6fqz\" (UniqueName: \"kubernetes.io/projected/e32e8e22-a821-4b4a-9f93-1983ebb326c0-kube-api-access-v6fqz\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:54.313269 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:54.313269 5274 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e32e8e22-a821-4b4a-9f93-1983ebb326c0-otlp-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:54.313469 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:54.313280 5274 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e32e8e22-a821-4b4a-9f93-1983ebb326c0-otc-internal\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:54.515845 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:42:54.515783 5274 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-usage-logs-collector-9d8884b6: configmap "data-science-usage-logs-collector-9d8884b6" not found Oct 31 00:42:54.515845 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:42:54.515831 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-usage-logs-collector-headless-tls: secret "data-science-usage-logs-collector-headless-tls" not found Oct 31 00:42:54.516053 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:42:54.515853 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/98049138-d8ad-41fc-9dc8-8347eb76f6c2-otc-internal podName:98049138-d8ad-41fc-9dc8-8347eb76f6c2 nodeName:}" failed. No retries permitted until 2026-10-31 00:42:55.515840181 +0000 UTC m=+2612.152658875 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "otc-internal" (UniqueName: "kubernetes.io/configmap/98049138-d8ad-41fc-9dc8-8347eb76f6c2-otc-internal") pod "data-science-usage-logs-collector-5b5b9f5c4f-6mgrd" (UID: "98049138-d8ad-41fc-9dc8-8347eb76f6c2") : configmap "data-science-usage-logs-collector-9d8884b6" not found Oct 31 00:42:54.516053 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:42:54.515895 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98049138-d8ad-41fc-9dc8-8347eb76f6c2-otlp-tls-certs podName:98049138-d8ad-41fc-9dc8-8347eb76f6c2 nodeName:}" failed. No retries permitted until 2026-10-31 00:42:55.515881749 +0000 UTC m=+2612.152700443 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/98049138-d8ad-41fc-9dc8-8347eb76f6c2-otlp-tls-certs") pod "data-science-usage-logs-collector-5b5b9f5c4f-6mgrd" (UID: "98049138-d8ad-41fc-9dc8-8347eb76f6c2") : secret "data-science-usage-logs-collector-headless-tls" not found Oct 31 00:42:54.856639 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:54.856559 5274 generic.go:356] "Generic (PLEG): container finished" podID="e32e8e22-a821-4b4a-9f93-1983ebb326c0" containerID="c648636f28bc746b12d62c8121756b4094c0c12e5ea4068b8162117ad68b7ff2" exitCode=0 Oct 31 00:42:54.856639 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:54.856618 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-ktqtv" Oct 31 00:42:54.856872 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:54.856655 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-ktqtv" event={"ID":"e32e8e22-a821-4b4a-9f93-1983ebb326c0","Type":"ContainerDied","Data":"c648636f28bc746b12d62c8121756b4094c0c12e5ea4068b8162117ad68b7ff2"} Oct 31 00:42:54.856872 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:54.856691 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-6mgrd" podUID="98049138-d8ad-41fc-9dc8-8347eb76f6c2" containerName="otc-container" containerID="cri-o://d240f5593abeca73523c81a2f4ab0ae8fcc12a83f2ad0a865207f5e3d059fbb6" gracePeriod=30 Oct 31 00:42:54.856872 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:54.856717 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-ktqtv" event={"ID":"e32e8e22-a821-4b4a-9f93-1983ebb326c0","Type":"ContainerDied","Data":"174bad2b2858ca821ef5ff9c48d462e665224164bb976698b0ffa791fb95ad2e"} Oct 31 00:42:54.856872 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:54.856746 5274 scope.go:122] "RemoveContainer" containerID="c648636f28bc746b12d62c8121756b4094c0c12e5ea4068b8162117ad68b7ff2" Oct 31 00:42:54.865906 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:54.865883 5274 scope.go:122] "RemoveContainer" containerID="c648636f28bc746b12d62c8121756b4094c0c12e5ea4068b8162117ad68b7ff2" Oct 31 00:42:54.866225 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:42:54.866200 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c648636f28bc746b12d62c8121756b4094c0c12e5ea4068b8162117ad68b7ff2\": container with ID starting with c648636f28bc746b12d62c8121756b4094c0c12e5ea4068b8162117ad68b7ff2 not found: ID does not exist" containerID="c648636f28bc746b12d62c8121756b4094c0c12e5ea4068b8162117ad68b7ff2" Oct 31 00:42:54.866292 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:54.866237 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c648636f28bc746b12d62c8121756b4094c0c12e5ea4068b8162117ad68b7ff2"} err="failed to get container status \"c648636f28bc746b12d62c8121756b4094c0c12e5ea4068b8162117ad68b7ff2\": rpc error: code = NotFound desc = could not find container \"c648636f28bc746b12d62c8121756b4094c0c12e5ea4068b8162117ad68b7ff2\": container with ID starting with c648636f28bc746b12d62c8121756b4094c0c12e5ea4068b8162117ad68b7ff2 not found: ID does not exist" Oct 31 00:42:54.892088 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:54.892068 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-ktqtv"] Oct 31 00:42:54.896126 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:54.896106 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-ktqtv"] Oct 31 00:42:55.107411 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:55.107357 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-6mgrd" Oct 31 00:42:55.221483 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:55.221457 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98049138-d8ad-41fc-9dc8-8347eb76f6c2-otc-internal\" (UniqueName: \"kubernetes.io/configmap/98049138-d8ad-41fc-9dc8-8347eb76f6c2-otc-internal\") pod \"98049138-d8ad-41fc-9dc8-8347eb76f6c2\" (UID: \"98049138-d8ad-41fc-9dc8-8347eb76f6c2\") " Oct 31 00:42:55.221798 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:55.221498 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98049138-d8ad-41fc-9dc8-8347eb76f6c2-kube-api-access-5sbgl\" (UniqueName: \"kubernetes.io/projected/98049138-d8ad-41fc-9dc8-8347eb76f6c2-kube-api-access-5sbgl\") pod \"98049138-d8ad-41fc-9dc8-8347eb76f6c2\" (UID: \"98049138-d8ad-41fc-9dc8-8347eb76f6c2\") " Oct 31 00:42:55.221798 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:55.221531 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98049138-d8ad-41fc-9dc8-8347eb76f6c2-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/98049138-d8ad-41fc-9dc8-8347eb76f6c2-otlp-tls-certs\") pod \"98049138-d8ad-41fc-9dc8-8347eb76f6c2\" (UID: \"98049138-d8ad-41fc-9dc8-8347eb76f6c2\") " Oct 31 00:42:55.221880 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:55.221834 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98049138-d8ad-41fc-9dc8-8347eb76f6c2-otc-internal" pod "98049138-d8ad-41fc-9dc8-8347eb76f6c2" (UID: "98049138-d8ad-41fc-9dc8-8347eb76f6c2"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:55.223602 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:55.223580 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98049138-d8ad-41fc-9dc8-8347eb76f6c2-otlp-tls-certs" pod "98049138-d8ad-41fc-9dc8-8347eb76f6c2" (UID: "98049138-d8ad-41fc-9dc8-8347eb76f6c2"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:55.223681 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:55.223635 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98049138-d8ad-41fc-9dc8-8347eb76f6c2-kube-api-access-5sbgl" pod "98049138-d8ad-41fc-9dc8-8347eb76f6c2" (UID: "98049138-d8ad-41fc-9dc8-8347eb76f6c2"). InnerVolumeSpecName "kube-api-access-5sbgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:55.322875 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:55.322840 5274 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/98049138-d8ad-41fc-9dc8-8347eb76f6c2-otc-internal\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:55.322875 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:55.322873 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sbgl\" (UniqueName: \"kubernetes.io/projected/98049138-d8ad-41fc-9dc8-8347eb76f6c2-kube-api-access-5sbgl\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:55.322875 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:55.322883 5274 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/98049138-d8ad-41fc-9dc8-8347eb76f6c2-otlp-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:55.480570 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:55.480545 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e32e8e22-a821-4b4a-9f93-1983ebb326c0" path="/var/lib/kubelet/pods/e32e8e22-a821-4b4a-9f93-1983ebb326c0/volumes" Oct 31 00:42:55.862363 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:55.862293 5274 generic.go:356] "Generic (PLEG): container finished" podID="98049138-d8ad-41fc-9dc8-8347eb76f6c2" containerID="d240f5593abeca73523c81a2f4ab0ae8fcc12a83f2ad0a865207f5e3d059fbb6" exitCode=0 Oct 31 00:42:55.862363 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:55.862338 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-6mgrd" Oct 31 00:42:55.862574 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:55.862369 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-6mgrd" event={"ID":"98049138-d8ad-41fc-9dc8-8347eb76f6c2","Type":"ContainerDied","Data":"d240f5593abeca73523c81a2f4ab0ae8fcc12a83f2ad0a865207f5e3d059fbb6"} Oct 31 00:42:55.862574 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:55.862420 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-6mgrd" event={"ID":"98049138-d8ad-41fc-9dc8-8347eb76f6c2","Type":"ContainerDied","Data":"8084974c1344b07307a04125b56c0609c4defded7014b4a5c967fcc92e9d4cec"} Oct 31 00:42:55.862574 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:55.862446 5274 scope.go:122] "RemoveContainer" containerID="d240f5593abeca73523c81a2f4ab0ae8fcc12a83f2ad0a865207f5e3d059fbb6" Oct 31 00:42:55.870985 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:55.870968 5274 scope.go:122] "RemoveContainer" containerID="d240f5593abeca73523c81a2f4ab0ae8fcc12a83f2ad0a865207f5e3d059fbb6" Oct 31 00:42:55.871250 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:42:55.871229 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d240f5593abeca73523c81a2f4ab0ae8fcc12a83f2ad0a865207f5e3d059fbb6\": container with ID starting with d240f5593abeca73523c81a2f4ab0ae8fcc12a83f2ad0a865207f5e3d059fbb6 not found: ID does not exist" containerID="d240f5593abeca73523c81a2f4ab0ae8fcc12a83f2ad0a865207f5e3d059fbb6" Oct 31 00:42:55.871300 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:55.871258 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d240f5593abeca73523c81a2f4ab0ae8fcc12a83f2ad0a865207f5e3d059fbb6"} err="failed to get container status \"d240f5593abeca73523c81a2f4ab0ae8fcc12a83f2ad0a865207f5e3d059fbb6\": rpc error: code = NotFound desc = could not find container \"d240f5593abeca73523c81a2f4ab0ae8fcc12a83f2ad0a865207f5e3d059fbb6\": container with ID starting with d240f5593abeca73523c81a2f4ab0ae8fcc12a83f2ad0a865207f5e3d059fbb6 not found: ID does not exist" Oct 31 00:42:55.879826 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:55.879804 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-6mgrd"] Oct 31 00:42:55.884605 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:55.884584 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-6mgrd"] Oct 31 00:42:57.480596 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:42:57.480565 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98049138-d8ad-41fc-9dc8-8347eb76f6c2" path="/var/lib/kubelet/pods/98049138-d8ad-41fc-9dc8-8347eb76f6c2/volumes" Oct 31 00:43:20.899113 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:43:20.899032 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:43:50.900480 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:43:50.900439 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:43:55.166597 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:55.166564 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tqskh"] Oct 31 00:43:55.167018 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:55.166895 5274 cpu_manager.go:414] "containerMap: removing container" podUID="98049138-d8ad-41fc-9dc8-8347eb76f6c2" containerName="otc-container" Oct 31 00:43:55.167018 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:55.166906 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98049138-d8ad-41fc-9dc8-8347eb76f6c2" containerName="otc-container" Oct 31 00:43:55.167018 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:55.166918 5274 cpu_manager.go:414] "containerMap: removing container" podUID="e32e8e22-a821-4b4a-9f93-1983ebb326c0" containerName="otc-container" Oct 31 00:43:55.167018 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:55.166923 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e32e8e22-a821-4b4a-9f93-1983ebb326c0" containerName="otc-container" Oct 31 00:43:55.167018 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:55.167008 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="98049138-d8ad-41fc-9dc8-8347eb76f6c2" containerName="otc-container" Oct 31 00:43:55.167018 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:55.167018 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="e32e8e22-a821-4b4a-9f93-1983ebb326c0" containerName="otc-container" Oct 31 00:43:55.170066 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:55.170043 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tqskh" Oct 31 00:43:55.180023 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:55.179912 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-headless-tls\"" Oct 31 00:43:55.180233 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:55.180067 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-9d8884b6\"" Oct 31 00:43:55.182463 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:55.182431 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tqskh"] Oct 31 00:43:55.228506 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:55.228464 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4br8\" (UniqueName: \"kubernetes.io/projected/cc153792-6589-4640-98dd-5930f1d0f6e4-kube-api-access-k4br8\") pod \"data-science-usage-logs-collector-5b5b9f5c4f-tqskh\" (UID: \"cc153792-6589-4640-98dd-5930f1d0f6e4\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tqskh" Oct 31 00:43:55.228506 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:55.228512 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cc153792-6589-4640-98dd-5930f1d0f6e4-otc-internal\") pod \"data-science-usage-logs-collector-5b5b9f5c4f-tqskh\" (UID: \"cc153792-6589-4640-98dd-5930f1d0f6e4\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tqskh" Oct 31 00:43:55.228762 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:55.228612 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc153792-6589-4640-98dd-5930f1d0f6e4-otlp-tls-certs\") pod \"data-science-usage-logs-collector-5b5b9f5c4f-tqskh\" (UID: \"cc153792-6589-4640-98dd-5930f1d0f6e4\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tqskh" Oct 31 00:43:55.329153 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:55.329109 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4br8\" (UniqueName: \"kubernetes.io/projected/cc153792-6589-4640-98dd-5930f1d0f6e4-kube-api-access-k4br8\") pod \"data-science-usage-logs-collector-5b5b9f5c4f-tqskh\" (UID: \"cc153792-6589-4640-98dd-5930f1d0f6e4\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tqskh" Oct 31 00:43:55.329153 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:55.329160 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cc153792-6589-4640-98dd-5930f1d0f6e4-otc-internal\") pod \"data-science-usage-logs-collector-5b5b9f5c4f-tqskh\" (UID: \"cc153792-6589-4640-98dd-5930f1d0f6e4\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tqskh" Oct 31 00:43:55.329433 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:55.329200 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc153792-6589-4640-98dd-5930f1d0f6e4-otlp-tls-certs\") pod \"data-science-usage-logs-collector-5b5b9f5c4f-tqskh\" (UID: \"cc153792-6589-4640-98dd-5930f1d0f6e4\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tqskh" Oct 31 00:43:55.329878 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:55.329845 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cc153792-6589-4640-98dd-5930f1d0f6e4-otc-internal\") pod \"data-science-usage-logs-collector-5b5b9f5c4f-tqskh\" (UID: \"cc153792-6589-4640-98dd-5930f1d0f6e4\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tqskh" Oct 31 00:43:55.331705 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:55.331680 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc153792-6589-4640-98dd-5930f1d0f6e4-otlp-tls-certs\") pod \"data-science-usage-logs-collector-5b5b9f5c4f-tqskh\" (UID: \"cc153792-6589-4640-98dd-5930f1d0f6e4\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tqskh" Oct 31 00:43:55.341851 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:55.341824 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4br8\" (UniqueName: \"kubernetes.io/projected/cc153792-6589-4640-98dd-5930f1d0f6e4-kube-api-access-k4br8\") pod \"data-science-usage-logs-collector-5b5b9f5c4f-tqskh\" (UID: \"cc153792-6589-4640-98dd-5930f1d0f6e4\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tqskh" Oct 31 00:43:55.487918 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:55.487838 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tqskh" Oct 31 00:43:55.769175 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:55.769146 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tqskh"] Oct 31 00:43:55.771826 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:43:55.771801 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc153792_6589_4640_98dd_5930f1d0f6e4.slice/crio-e5eb36d5f8089a80aeda193a890f20ee9e388b8a5f0333c0539b6d90cb5c48c2 WatchSource:0}: Error finding container e5eb36d5f8089a80aeda193a890f20ee9e388b8a5f0333c0539b6d90cb5c48c2: Status 404 returned error can't find the container with id e5eb36d5f8089a80aeda193a890f20ee9e388b8a5f0333c0539b6d90cb5c48c2 Oct 31 00:43:55.773841 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:55.773810 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:43:55.773914 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:55.773872 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:43:56.155291 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:56.155257 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tqskh" event={"ID":"cc153792-6589-4640-98dd-5930f1d0f6e4","Type":"ContainerStarted","Data":"95a02c35238e095c787b5aad5b604a7c6176e2d9381678526b66e7137a00415b"} Oct 31 00:43:56.155483 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:56.155311 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tqskh" event={"ID":"cc153792-6589-4640-98dd-5930f1d0f6e4","Type":"ContainerStarted","Data":"e5eb36d5f8089a80aeda193a890f20ee9e388b8a5f0333c0539b6d90cb5c48c2"} Oct 31 00:43:56.176289 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:56.176234 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tqskh" podStartSLOduration=1.176219973 podStartE2EDuration="1.176219973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:56.174992523 +0000 UTC m=+2672.811811240" watchObservedRunningTime="2026-10-31 00:43:56.176219973 +0000 UTC m=+2672.813038755" Oct 31 00:43:56.872936 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:56.872901 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f"] Oct 31 00:43:56.873172 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:56.873142 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" podUID="62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1" containerName="gateway" containerID="cri-o://5399d3fd294a4dc2f657ae57cd261f7344867a6ef267ddfef5c3f775321d9a31" gracePeriod=30 Oct 31 00:43:56.873244 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:56.873172 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" podUID="62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1" containerName="opa" containerID="cri-o://7ea1134091db4eabca6878ea9f1c3c0d5d7a7b30ed3e2ffceaf2ea6fc149c7dd" gracePeriod=30 Oct 31 00:43:56.905103 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:56.905071 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tqskh"] Oct 31 00:43:57.162354 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:57.162267 5274 generic.go:356] "Generic (PLEG): container finished" podID="62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1" containerID="7ea1134091db4eabca6878ea9f1c3c0d5d7a7b30ed3e2ffceaf2ea6fc149c7dd" exitCode=0 Oct 31 00:43:57.162354 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:57.162341 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" event={"ID":"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1","Type":"ContainerDied","Data":"7ea1134091db4eabca6878ea9f1c3c0d5d7a7b30ed3e2ffceaf2ea6fc149c7dd"} Oct 31 00:43:57.246257 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:43:57.246218 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-usage-logs-collector-headless-tls: secret "data-science-usage-logs-collector-headless-tls" not found Oct 31 00:43:57.246257 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:43:57.246257 5274 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-usage-logs-collector-9d8884b6: configmap "data-science-usage-logs-collector-9d8884b6" not found Oct 31 00:43:57.246655 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:43:57.246298 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc153792-6589-4640-98dd-5930f1d0f6e4-otlp-tls-certs podName:cc153792-6589-4640-98dd-5930f1d0f6e4 nodeName:}" failed. No retries permitted until 2026-10-31 00:43:57.746281415 +0000 UTC m=+2674.383100109 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/cc153792-6589-4640-98dd-5930f1d0f6e4-otlp-tls-certs") pod "data-science-usage-logs-collector-5b5b9f5c4f-tqskh" (UID: "cc153792-6589-4640-98dd-5930f1d0f6e4") : secret "data-science-usage-logs-collector-headless-tls" not found Oct 31 00:43:57.246655 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:43:57.246314 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc153792-6589-4640-98dd-5930f1d0f6e4-otc-internal podName:cc153792-6589-4640-98dd-5930f1d0f6e4 nodeName:}" failed. No retries permitted until 2026-10-31 00:43:57.74630824 +0000 UTC m=+2674.383126934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otc-internal" (UniqueName: "kubernetes.io/configmap/cc153792-6589-4640-98dd-5930f1d0f6e4-otc-internal") pod "data-science-usage-logs-collector-5b5b9f5c4f-tqskh" (UID: "cc153792-6589-4640-98dd-5930f1d0f6e4") : configmap "data-science-usage-logs-collector-9d8884b6" not found Oct 31 00:43:57.750947 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:43:57.750915 5274 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-usage-logs-collector-9d8884b6: configmap "data-science-usage-logs-collector-9d8884b6" not found Oct 31 00:43:57.751128 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:43:57.750999 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc153792-6589-4640-98dd-5930f1d0f6e4-otc-internal podName:cc153792-6589-4640-98dd-5930f1d0f6e4 nodeName:}" failed. No retries permitted until 2026-10-31 00:43:58.750983027 +0000 UTC m=+2675.387801721 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "otc-internal" (UniqueName: "kubernetes.io/configmap/cc153792-6589-4640-98dd-5930f1d0f6e4-otc-internal") pod "data-science-usage-logs-collector-5b5b9f5c4f-tqskh" (UID: "cc153792-6589-4640-98dd-5930f1d0f6e4") : configmap "data-science-usage-logs-collector-9d8884b6" not found Oct 31 00:43:57.751128 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:43:57.751019 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-usage-logs-collector-headless-tls: secret "data-science-usage-logs-collector-headless-tls" not found Oct 31 00:43:57.751128 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:43:57.751068 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc153792-6589-4640-98dd-5930f1d0f6e4-otlp-tls-certs podName:cc153792-6589-4640-98dd-5930f1d0f6e4 nodeName:}" failed. No retries permitted until 2026-10-31 00:43:58.751056987 +0000 UTC m=+2675.387875681 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/cc153792-6589-4640-98dd-5930f1d0f6e4-otlp-tls-certs") pod "data-science-usage-logs-collector-5b5b9f5c4f-tqskh" (UID: "cc153792-6589-4640-98dd-5930f1d0f6e4") : secret "data-science-usage-logs-collector-headless-tls" not found Oct 31 00:43:58.120644 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.120612 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" Oct 31 00:43:58.154488 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.154455 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-rbac\" (UniqueName: \"kubernetes.io/configmap/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-rbac\") pod \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\" (UID: \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\") " Oct 31 00:43:58.154488 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.154499 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-lokistack-gateway\") pod \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\" (UID: \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\") " Oct 31 00:43:58.154760 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.154522 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-data-science-usage-logs-lokistack-ca-bundle\") pod \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\" (UID: \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\") " Oct 31 00:43:58.154760 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.154549 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-tenants\" (UniqueName: \"kubernetes.io/secret/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-tenants\") pod \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\" (UID: \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\") " Oct 31 00:43:58.154760 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.154613 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-kube-api-access-h9dgx\" (UniqueName: \"kubernetes.io/projected/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-kube-api-access-h9dgx\") pod \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\" (UID: \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\") " Oct 31 00:43:58.154760 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.154650 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-data-science-usage-logs-lokistack-gateway-client-http\") pod \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\" (UID: \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\") " Oct 31 00:43:58.154760 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.154683 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-tls-secret\" (UniqueName: \"kubernetes.io/projected/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-tls-secret\") pod \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\" (UID: \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\") " Oct 31 00:43:58.154760 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.154719 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\" (UID: \"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1\") " Oct 31 00:43:58.155093 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.154969 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-data-science-usage-logs-lokistack-ca-bundle" pod "62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1" (UID: "62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:58.155155 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.155093 5274 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:58.157246 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.157212 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-kube-api-access-h9dgx" pod "62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1" (UID: "62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1"). InnerVolumeSpecName "kube-api-access-h9dgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:58.157506 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.157465 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-data-science-usage-logs-lokistack-gateway-ca-bundle" pod "62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1" (UID: "62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1"). InnerVolumeSpecName "data-science-usage-logs-lokistack-gateway-ca-bundle". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:58.157776 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.157743 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-data-science-usage-logs-lokistack-gateway-client-http" pod "62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1" (UID: "62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1"). InnerVolumeSpecName "data-science-usage-logs-lokistack-gateway-client-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:58.157984 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.157937 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-tls-secret" pod "62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1" (UID: "62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1"). InnerVolumeSpecName "tls-secret". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:58.160359 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.160333 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-rbac" pod "62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1" (UID: "62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:58.161431 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.161384 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-lokistack-gateway" pod "62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1" (UID: "62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1"). InnerVolumeSpecName "lokistack-gateway". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:58.161977 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.161932 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-tenants" pod "62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1" (UID: "62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1"). InnerVolumeSpecName "tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:58.170905 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.170873 5274 generic.go:356] "Generic (PLEG): container finished" podID="62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1" containerID="5399d3fd294a4dc2f657ae57cd261f7344867a6ef267ddfef5c3f775321d9a31" exitCode=0 Oct 31 00:43:58.171071 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.170917 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" Oct 31 00:43:58.171071 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.170979 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" event={"ID":"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1","Type":"ContainerDied","Data":"5399d3fd294a4dc2f657ae57cd261f7344867a6ef267ddfef5c3f775321d9a31"} Oct 31 00:43:58.171071 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.171034 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f" event={"ID":"62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1","Type":"ContainerDied","Data":"1751a2a28dea5eedb105827872756754414e4c8ff1502414430a8d73582152ba"} Oct 31 00:43:58.171071 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.171059 5274 scope.go:122] "RemoveContainer" containerID="7ea1134091db4eabca6878ea9f1c3c0d5d7a7b30ed3e2ffceaf2ea6fc149c7dd" Oct 31 00:43:58.171466 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.171284 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tqskh" podUID="cc153792-6589-4640-98dd-5930f1d0f6e4" containerName="otc-container" containerID="cri-o://95a02c35238e095c787b5aad5b604a7c6176e2d9381678526b66e7137a00415b" gracePeriod=30 Oct 31 00:43:58.180898 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.180871 5274 scope.go:122] "RemoveContainer" containerID="5399d3fd294a4dc2f657ae57cd261f7344867a6ef267ddfef5c3f775321d9a31" Oct 31 00:43:58.189910 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.189887 5274 scope.go:122] "RemoveContainer" containerID="7ea1134091db4eabca6878ea9f1c3c0d5d7a7b30ed3e2ffceaf2ea6fc149c7dd" Oct 31 00:43:58.190221 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:43:58.190202 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ea1134091db4eabca6878ea9f1c3c0d5d7a7b30ed3e2ffceaf2ea6fc149c7dd\": container with ID starting with 7ea1134091db4eabca6878ea9f1c3c0d5d7a7b30ed3e2ffceaf2ea6fc149c7dd not found: ID does not exist" containerID="7ea1134091db4eabca6878ea9f1c3c0d5d7a7b30ed3e2ffceaf2ea6fc149c7dd" Oct 31 00:43:58.190317 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.190230 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ea1134091db4eabca6878ea9f1c3c0d5d7a7b30ed3e2ffceaf2ea6fc149c7dd"} err="failed to get container status \"7ea1134091db4eabca6878ea9f1c3c0d5d7a7b30ed3e2ffceaf2ea6fc149c7dd\": rpc error: code = NotFound desc = could not find container \"7ea1134091db4eabca6878ea9f1c3c0d5d7a7b30ed3e2ffceaf2ea6fc149c7dd\": container with ID starting with 7ea1134091db4eabca6878ea9f1c3c0d5d7a7b30ed3e2ffceaf2ea6fc149c7dd not found: ID does not exist" Oct 31 00:43:58.190317 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.190248 5274 scope.go:122] "RemoveContainer" containerID="5399d3fd294a4dc2f657ae57cd261f7344867a6ef267ddfef5c3f775321d9a31" Oct 31 00:43:58.190533 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:43:58.190454 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5399d3fd294a4dc2f657ae57cd261f7344867a6ef267ddfef5c3f775321d9a31\": container with ID starting with 5399d3fd294a4dc2f657ae57cd261f7344867a6ef267ddfef5c3f775321d9a31 not found: ID does not exist" containerID="5399d3fd294a4dc2f657ae57cd261f7344867a6ef267ddfef5c3f775321d9a31" Oct 31 00:43:58.190533 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.190470 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5399d3fd294a4dc2f657ae57cd261f7344867a6ef267ddfef5c3f775321d9a31"} err="failed to get container status \"5399d3fd294a4dc2f657ae57cd261f7344867a6ef267ddfef5c3f775321d9a31\": rpc error: code = NotFound desc = could not find container \"5399d3fd294a4dc2f657ae57cd261f7344867a6ef267ddfef5c3f775321d9a31\": container with ID starting with 5399d3fd294a4dc2f657ae57cd261f7344867a6ef267ddfef5c3f775321d9a31 not found: ID does not exist" Oct 31 00:43:58.212598 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.212566 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f"] Oct 31 00:43:58.217196 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.217170 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-6856ccc797-gdm8f"] Oct 31 00:43:58.256437 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.256398 5274 reconciler_common.go:299] "Volume detached for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-tenants\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:58.256437 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.256427 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9dgx\" (UniqueName: \"kubernetes.io/projected/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-kube-api-access-h9dgx\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:58.256437 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.256438 5274 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-data-science-usage-logs-lokistack-gateway-client-http\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:58.256881 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.256449 5274 reconciler_common.go:299] "Volume detached for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-tls-secret\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:58.256881 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.256459 5274 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-data-science-usage-logs-lokistack-gateway-ca-bundle\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:58.256881 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.256468 5274 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-rbac\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:58.256881 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.256476 5274 reconciler_common.go:299] "Volume detached for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1-lokistack-gateway\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:58.406853 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.406827 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tqskh" Oct 31 00:43:58.458360 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.458321 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc153792-6589-4640-98dd-5930f1d0f6e4-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc153792-6589-4640-98dd-5930f1d0f6e4-otlp-tls-certs\") pod \"cc153792-6589-4640-98dd-5930f1d0f6e4\" (UID: \"cc153792-6589-4640-98dd-5930f1d0f6e4\") " Oct 31 00:43:58.458558 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.458421 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc153792-6589-4640-98dd-5930f1d0f6e4-otc-internal\" (UniqueName: \"kubernetes.io/configmap/cc153792-6589-4640-98dd-5930f1d0f6e4-otc-internal\") pod \"cc153792-6589-4640-98dd-5930f1d0f6e4\" (UID: \"cc153792-6589-4640-98dd-5930f1d0f6e4\") " Oct 31 00:43:58.458558 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.458455 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc153792-6589-4640-98dd-5930f1d0f6e4-kube-api-access-k4br8\" (UniqueName: \"kubernetes.io/projected/cc153792-6589-4640-98dd-5930f1d0f6e4-kube-api-access-k4br8\") pod \"cc153792-6589-4640-98dd-5930f1d0f6e4\" (UID: \"cc153792-6589-4640-98dd-5930f1d0f6e4\") " Oct 31 00:43:58.458826 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.458788 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc153792-6589-4640-98dd-5930f1d0f6e4-otc-internal" pod "cc153792-6589-4640-98dd-5930f1d0f6e4" (UID: "cc153792-6589-4640-98dd-5930f1d0f6e4"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:58.460556 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.460534 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc153792-6589-4640-98dd-5930f1d0f6e4-kube-api-access-k4br8" pod "cc153792-6589-4640-98dd-5930f1d0f6e4" (UID: "cc153792-6589-4640-98dd-5930f1d0f6e4"). InnerVolumeSpecName "kube-api-access-k4br8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:58.460645 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.460539 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc153792-6589-4640-98dd-5930f1d0f6e4-otlp-tls-certs" pod "cc153792-6589-4640-98dd-5930f1d0f6e4" (UID: "cc153792-6589-4640-98dd-5930f1d0f6e4"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:58.560031 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.559927 5274 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc153792-6589-4640-98dd-5930f1d0f6e4-otlp-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:58.560031 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.559979 5274 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cc153792-6589-4640-98dd-5930f1d0f6e4-otc-internal\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:58.560031 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:58.559995 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4br8\" (UniqueName: \"kubernetes.io/projected/cc153792-6589-4640-98dd-5930f1d0f6e4-kube-api-access-k4br8\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:59.178209 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:59.178174 5274 generic.go:356] "Generic (PLEG): container finished" podID="cc153792-6589-4640-98dd-5930f1d0f6e4" containerID="95a02c35238e095c787b5aad5b604a7c6176e2d9381678526b66e7137a00415b" exitCode=0 Oct 31 00:43:59.178392 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:59.178218 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tqskh" Oct 31 00:43:59.178392 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:59.178253 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tqskh" event={"ID":"cc153792-6589-4640-98dd-5930f1d0f6e4","Type":"ContainerDied","Data":"95a02c35238e095c787b5aad5b604a7c6176e2d9381678526b66e7137a00415b"} Oct 31 00:43:59.178392 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:59.178305 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tqskh" event={"ID":"cc153792-6589-4640-98dd-5930f1d0f6e4","Type":"ContainerDied","Data":"e5eb36d5f8089a80aeda193a890f20ee9e388b8a5f0333c0539b6d90cb5c48c2"} Oct 31 00:43:59.178392 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:59.178329 5274 scope.go:122] "RemoveContainer" containerID="95a02c35238e095c787b5aad5b604a7c6176e2d9381678526b66e7137a00415b" Oct 31 00:43:59.187051 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:59.187029 5274 scope.go:122] "RemoveContainer" containerID="95a02c35238e095c787b5aad5b604a7c6176e2d9381678526b66e7137a00415b" Oct 31 00:43:59.187344 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:43:59.187324 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95a02c35238e095c787b5aad5b604a7c6176e2d9381678526b66e7137a00415b\": container with ID starting with 95a02c35238e095c787b5aad5b604a7c6176e2d9381678526b66e7137a00415b not found: ID does not exist" containerID="95a02c35238e095c787b5aad5b604a7c6176e2d9381678526b66e7137a00415b" Oct 31 00:43:59.187417 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:59.187351 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95a02c35238e095c787b5aad5b604a7c6176e2d9381678526b66e7137a00415b"} err="failed to get container status \"95a02c35238e095c787b5aad5b604a7c6176e2d9381678526b66e7137a00415b\": rpc error: code = NotFound desc = could not find container \"95a02c35238e095c787b5aad5b604a7c6176e2d9381678526b66e7137a00415b\": container with ID starting with 95a02c35238e095c787b5aad5b604a7c6176e2d9381678526b66e7137a00415b not found: ID does not exist" Oct 31 00:43:59.201308 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:59.201235 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tqskh"] Oct 31 00:43:59.205991 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:59.205932 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tqskh"] Oct 31 00:43:59.481103 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:59.481030 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1" path="/var/lib/kubelet/pods/62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1/volumes" Oct 31 00:43:59.481461 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:43:59.481449 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc153792-6589-4640-98dd-5930f1d0f6e4" path="/var/lib/kubelet/pods/cc153792-6589-4640-98dd-5930f1d0f6e4/volumes" Oct 31 00:44:00.975600 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:00.975568 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql"] Oct 31 00:44:00.976073 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:00.976057 5274 cpu_manager.go:414] "containerMap: removing container" podUID="62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1" containerName="opa" Oct 31 00:44:00.976142 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:00.976077 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1" containerName="opa" Oct 31 00:44:00.976142 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:00.976113 5274 cpu_manager.go:414] "containerMap: removing container" podUID="62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1" containerName="gateway" Oct 31 00:44:00.976142 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:00.976121 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1" containerName="gateway" Oct 31 00:44:00.976261 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:00.976148 5274 cpu_manager.go:414] "containerMap: removing container" podUID="cc153792-6589-4640-98dd-5930f1d0f6e4" containerName="otc-container" Oct 31 00:44:00.976261 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:00.976156 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc153792-6589-4640-98dd-5930f1d0f6e4" containerName="otc-container" Oct 31 00:44:00.976261 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:00.976243 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc153792-6589-4640-98dd-5930f1d0f6e4" containerName="otc-container" Oct 31 00:44:00.976261 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:00.976257 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1" containerName="opa" Oct 31 00:44:00.976394 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:00.976269 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="62c0bd2d-d4e2-49c0-9cf6-79acc1364bb1" containerName="gateway" Oct 31 00:44:00.980970 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:00.980934 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" Oct 31 00:44:00.985787 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:00.985759 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway\"" Oct 31 00:44:00.989303 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:00.989280 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:44:00.989556 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:00.989530 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-ca-bundle\"" Oct 31 00:44:00.989556 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:00.989552 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:44:00.991507 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:00.991486 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway-ca-bundle\"" Oct 31 00:44:00.992036 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:00.992014 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway\"" Oct 31 00:44:00.992391 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:00.992359 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway-http\"" Oct 31 00:44:00.992650 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:00.992633 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway-client-http\"" Oct 31 00:44:00.994226 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:00.994200 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql"] Oct 31 00:44:01.081449 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:01.081413 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/0a9115f6-051d-42da-a1b3-a59cf69d0b03-data-science-usage-logs-lokistack-gateway-client-http\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql\" (UID: \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" Oct 31 00:44:01.081449 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:01.081450 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-899ww\" (UniqueName: \"kubernetes.io/projected/0a9115f6-051d-42da-a1b3-a59cf69d0b03-kube-api-access-899ww\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql\" (UID: \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" Oct 31 00:44:01.081701 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:01.081476 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/0a9115f6-051d-42da-a1b3-a59cf69d0b03-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql\" (UID: \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" Oct 31 00:44:01.081701 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:01.081599 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/0a9115f6-051d-42da-a1b3-a59cf69d0b03-tenants\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql\" (UID: \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" Oct 31 00:44:01.081701 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:01.081654 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/0a9115f6-051d-42da-a1b3-a59cf69d0b03-lokistack-gateway\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql\" (UID: \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" Oct 31 00:44:01.081863 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:01.081714 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/0a9115f6-051d-42da-a1b3-a59cf69d0b03-tls-secret\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql\" (UID: \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" Oct 31 00:44:01.081863 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:01.081739 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/0a9115f6-051d-42da-a1b3-a59cf69d0b03-rbac\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql\" (UID: \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" Oct 31 00:44:01.081863 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:01.081822 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a9115f6-051d-42da-a1b3-a59cf69d0b03-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql\" (UID: \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" Oct 31 00:44:01.183097 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:01.183063 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/0a9115f6-051d-42da-a1b3-a59cf69d0b03-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql\" (UID: \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" Oct 31 00:44:01.183321 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:01.183106 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/0a9115f6-051d-42da-a1b3-a59cf69d0b03-tenants\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql\" (UID: \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" Oct 31 00:44:01.183321 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:01.183233 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/0a9115f6-051d-42da-a1b3-a59cf69d0b03-lokistack-gateway\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql\" (UID: \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" Oct 31 00:44:01.183321 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:01.183279 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/0a9115f6-051d-42da-a1b3-a59cf69d0b03-tls-secret\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql\" (UID: \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" Oct 31 00:44:01.183321 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:01.183297 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/0a9115f6-051d-42da-a1b3-a59cf69d0b03-rbac\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql\" (UID: \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" Oct 31 00:44:01.183321 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:01.183319 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a9115f6-051d-42da-a1b3-a59cf69d0b03-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql\" (UID: \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" Oct 31 00:44:01.183590 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:01.183357 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/0a9115f6-051d-42da-a1b3-a59cf69d0b03-data-science-usage-logs-lokistack-gateway-client-http\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql\" (UID: \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" Oct 31 00:44:01.183590 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:01.183374 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-899ww\" (UniqueName: \"kubernetes.io/projected/0a9115f6-051d-42da-a1b3-a59cf69d0b03-kube-api-access-899ww\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql\" (UID: \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" Oct 31 00:44:01.184185 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:01.184153 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/0a9115f6-051d-42da-a1b3-a59cf69d0b03-lokistack-gateway\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql\" (UID: \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" Oct 31 00:44:01.184185 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:01.184169 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a9115f6-051d-42da-a1b3-a59cf69d0b03-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql\" (UID: \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" Oct 31 00:44:01.184840 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:01.184233 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/0a9115f6-051d-42da-a1b3-a59cf69d0b03-rbac\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql\" (UID: \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" Oct 31 00:44:01.186052 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:01.186024 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/0a9115f6-051d-42da-a1b3-a59cf69d0b03-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql\" (UID: \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" Oct 31 00:44:01.186168 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:01.186072 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/0a9115f6-051d-42da-a1b3-a59cf69d0b03-tenants\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql\" (UID: \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" Oct 31 00:44:01.186235 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:01.186175 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/0a9115f6-051d-42da-a1b3-a59cf69d0b03-tls-secret\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql\" (UID: \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" Oct 31 00:44:01.186443 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:01.186425 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/0a9115f6-051d-42da-a1b3-a59cf69d0b03-data-science-usage-logs-lokistack-gateway-client-http\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql\" (UID: \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" Oct 31 00:44:01.197836 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:01.197807 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-899ww\" (UniqueName: \"kubernetes.io/projected/0a9115f6-051d-42da-a1b3-a59cf69d0b03-kube-api-access-899ww\") pod \"data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql\" (UID: \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" Oct 31 00:44:01.297656 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:01.297566 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" Oct 31 00:44:01.580455 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:01.580426 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql"] Oct 31 00:44:01.582870 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:44:01.582841 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a9115f6_051d_42da_a1b3_a59cf69d0b03.slice/crio-eb57543f5587d4b16c643d278f8c67c311a471d375a9e1a4b6541db422752ce0 WatchSource:0}: Error finding container eb57543f5587d4b16c643d278f8c67c311a471d375a9e1a4b6541db422752ce0: Status 404 returned error can't find the container with id eb57543f5587d4b16c643d278f8c67c311a471d375a9e1a4b6541db422752ce0 Oct 31 00:44:02.198014 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:02.197971 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" event={"ID":"0a9115f6-051d-42da-a1b3-a59cf69d0b03","Type":"ContainerStarted","Data":"e8612cad31b72fed8ba75ce1e73f56f05069a38d845d3b06dfb851d51b0ab291"} Oct 31 00:44:02.198408 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:02.198029 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" event={"ID":"0a9115f6-051d-42da-a1b3-a59cf69d0b03","Type":"ContainerStarted","Data":"0902458c8fe7a673ada51d8ed2f67d19ab310f56824202b4bff5a2526caab800"} Oct 31 00:44:02.198408 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:02.198046 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" event={"ID":"0a9115f6-051d-42da-a1b3-a59cf69d0b03","Type":"ContainerStarted","Data":"eb57543f5587d4b16c643d278f8c67c311a471d375a9e1a4b6541db422752ce0"} Oct 31 00:44:02.198408 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:02.198310 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" Oct 31 00:44:02.204280 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:02.204260 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" Oct 31 00:44:02.223035 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:02.222993 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" podStartSLOduration=2.222979715 podStartE2EDuration="2.222979715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:02.221558964 +0000 UTC m=+2678.858377681" watchObservedRunningTime="2026-10-31 00:44:02.222979715 +0000 UTC m=+2678.859798422" Oct 31 00:44:03.202594 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:03.202561 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" Oct 31 00:44:03.210472 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:44:03.210447 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" Oct 31 00:44:20.899029 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:44:20.898997 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:44:50.899381 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:44:50.899340 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:45:00.363796 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:00.363762 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-swvbn"] Oct 31 00:45:00.367237 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:00.367213 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-swvbn" Oct 31 00:45:00.382207 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:00.382175 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-headless-tls\"" Oct 31 00:45:00.382572 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:00.382370 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-9d8884b6\"" Oct 31 00:45:00.383025 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:00.383007 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-swvbn"] Oct 31 00:45:00.442890 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:00.442859 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/37a5fadd-c499-4540-b603-1fb4b01821d1-otlp-tls-certs\") pod \"data-science-usage-logs-collector-76bd85dc9-swvbn\" (UID: \"37a5fadd-c499-4540-b603-1fb4b01821d1\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-swvbn" Oct 31 00:45:00.442890 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:00.442891 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxm2r\" (UniqueName: \"kubernetes.io/projected/37a5fadd-c499-4540-b603-1fb4b01821d1-kube-api-access-pxm2r\") pod \"data-science-usage-logs-collector-76bd85dc9-swvbn\" (UID: \"37a5fadd-c499-4540-b603-1fb4b01821d1\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-swvbn" Oct 31 00:45:00.443088 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:00.442923 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/37a5fadd-c499-4540-b603-1fb4b01821d1-otc-internal\") pod \"data-science-usage-logs-collector-76bd85dc9-swvbn\" (UID: \"37a5fadd-c499-4540-b603-1fb4b01821d1\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-swvbn" Oct 31 00:45:00.544035 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:00.544005 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/37a5fadd-c499-4540-b603-1fb4b01821d1-otlp-tls-certs\") pod \"data-science-usage-logs-collector-76bd85dc9-swvbn\" (UID: \"37a5fadd-c499-4540-b603-1fb4b01821d1\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-swvbn" Oct 31 00:45:00.544207 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:00.544043 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxm2r\" (UniqueName: \"kubernetes.io/projected/37a5fadd-c499-4540-b603-1fb4b01821d1-kube-api-access-pxm2r\") pod \"data-science-usage-logs-collector-76bd85dc9-swvbn\" (UID: \"37a5fadd-c499-4540-b603-1fb4b01821d1\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-swvbn" Oct 31 00:45:00.544207 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:00.544154 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/37a5fadd-c499-4540-b603-1fb4b01821d1-otc-internal\") pod \"data-science-usage-logs-collector-76bd85dc9-swvbn\" (UID: \"37a5fadd-c499-4540-b603-1fb4b01821d1\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-swvbn" Oct 31 00:45:00.545862 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:00.545829 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/37a5fadd-c499-4540-b603-1fb4b01821d1-otc-internal\") pod \"data-science-usage-logs-collector-76bd85dc9-swvbn\" (UID: \"37a5fadd-c499-4540-b603-1fb4b01821d1\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-swvbn" Oct 31 00:45:00.552243 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:00.552216 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/37a5fadd-c499-4540-b603-1fb4b01821d1-otlp-tls-certs\") pod \"data-science-usage-logs-collector-76bd85dc9-swvbn\" (UID: \"37a5fadd-c499-4540-b603-1fb4b01821d1\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-swvbn" Oct 31 00:45:00.556396 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:00.556376 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxm2r\" (UniqueName: \"kubernetes.io/projected/37a5fadd-c499-4540-b603-1fb4b01821d1-kube-api-access-pxm2r\") pod \"data-science-usage-logs-collector-76bd85dc9-swvbn\" (UID: \"37a5fadd-c499-4540-b603-1fb4b01821d1\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-swvbn" Oct 31 00:45:00.681474 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:00.681450 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-swvbn" Oct 31 00:45:00.952902 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:00.952876 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-swvbn"] Oct 31 00:45:00.955497 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:45:00.955460 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37a5fadd_c499_4540_b603_1fb4b01821d1.slice/crio-7d72bca99057ac9c1a822b0ea731cc491b767d5e03a910a0af3cd6f4a37070fa WatchSource:0}: Error finding container 7d72bca99057ac9c1a822b0ea731cc491b767d5e03a910a0af3cd6f4a37070fa: Status 404 returned error can't find the container with id 7d72bca99057ac9c1a822b0ea731cc491b767d5e03a910a0af3cd6f4a37070fa Oct 31 00:45:00.957745 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:00.957710 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:45:00.957841 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:00.957782 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:45:01.481650 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:01.481611 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-swvbn" event={"ID":"37a5fadd-c499-4540-b603-1fb4b01821d1","Type":"ContainerStarted","Data":"60c55da15380287f4d1fa657d48582abf54fe0986498eb8966287e3e041dcf6b"} Oct 31 00:45:01.482072 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:01.481668 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-swvbn" event={"ID":"37a5fadd-c499-4540-b603-1fb4b01821d1","Type":"ContainerStarted","Data":"7d72bca99057ac9c1a822b0ea731cc491b767d5e03a910a0af3cd6f4a37070fa"} Oct 31 00:45:01.499759 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:01.499712 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-swvbn" podStartSLOduration=1.4996991849999999 podStartE2EDuration="1.499699185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:01.499537448 +0000 UTC m=+2738.136356164" watchObservedRunningTime="2026-10-31 00:45:01.499699185 +0000 UTC m=+2738.136517901" Oct 31 00:45:02.563070 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:45:02.563031 5274 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-usage-logs-collector-9d8884b6: configmap "data-science-usage-logs-collector-9d8884b6" not found Oct 31 00:45:02.563647 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:45:02.563625 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/37a5fadd-c499-4540-b603-1fb4b01821d1-otc-internal podName:37a5fadd-c499-4540-b603-1fb4b01821d1 nodeName:}" failed. No retries permitted until 2026-10-31 00:45:03.063599062 +0000 UTC m=+2739.700417760 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otc-internal" (UniqueName: "kubernetes.io/configmap/37a5fadd-c499-4540-b603-1fb4b01821d1-otc-internal") pod "data-science-usage-logs-collector-76bd85dc9-swvbn" (UID: "37a5fadd-c499-4540-b603-1fb4b01821d1") : configmap "data-science-usage-logs-collector-9d8884b6" not found Oct 31 00:45:02.607313 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:02.607281 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql"] Oct 31 00:45:02.607649 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:02.607612 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" podUID="0a9115f6-051d-42da-a1b3-a59cf69d0b03" containerName="gateway" containerID="cri-o://0902458c8fe7a673ada51d8ed2f67d19ab310f56824202b4bff5a2526caab800" gracePeriod=30 Oct 31 00:45:02.608598 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:02.607701 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" podUID="0a9115f6-051d-42da-a1b3-a59cf69d0b03" containerName="opa" containerID="cri-o://e8612cad31b72fed8ba75ce1e73f56f05069a38d845d3b06dfb851d51b0ab291" gracePeriod=30 Oct 31 00:45:02.610699 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:02.610675 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-swvbn"] Oct 31 00:45:03.067057 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:45:03.067028 5274 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-usage-logs-collector-9d8884b6: configmap "data-science-usage-logs-collector-9d8884b6" not found Oct 31 00:45:03.067242 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:45:03.067095 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/37a5fadd-c499-4540-b603-1fb4b01821d1-otc-internal podName:37a5fadd-c499-4540-b603-1fb4b01821d1 nodeName:}" failed. No retries permitted until 2026-10-31 00:45:04.067078851 +0000 UTC m=+2740.703897548 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "otc-internal" (UniqueName: "kubernetes.io/configmap/37a5fadd-c499-4540-b603-1fb4b01821d1-otc-internal") pod "data-science-usage-logs-collector-76bd85dc9-swvbn" (UID: "37a5fadd-c499-4540-b603-1fb4b01821d1") : configmap "data-science-usage-logs-collector-9d8884b6" not found Oct 31 00:45:03.206809 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:03.206780 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" podUID="0a9115f6-051d-42da-a1b3-a59cf69d0b03" containerName="gateway" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:45:03.495043 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:03.495015 5274 generic.go:356] "Generic (PLEG): container finished" podID="0a9115f6-051d-42da-a1b3-a59cf69d0b03" containerID="e8612cad31b72fed8ba75ce1e73f56f05069a38d845d3b06dfb851d51b0ab291" exitCode=0 Oct 31 00:45:03.495202 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:03.495087 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" event={"ID":"0a9115f6-051d-42da-a1b3-a59cf69d0b03","Type":"ContainerDied","Data":"e8612cad31b72fed8ba75ce1e73f56f05069a38d845d3b06dfb851d51b0ab291"} Oct 31 00:45:03.495202 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:03.495172 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-swvbn" podUID="37a5fadd-c499-4540-b603-1fb4b01821d1" containerName="otc-container" containerID="cri-o://60c55da15380287f4d1fa657d48582abf54fe0986498eb8966287e3e041dcf6b" gracePeriod=30 Oct 31 00:45:03.768725 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:03.768703 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-swvbn" Oct 31 00:45:03.860865 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:03.860841 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" Oct 31 00:45:03.872931 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:03.872909 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37a5fadd-c499-4540-b603-1fb4b01821d1-kube-api-access-pxm2r\" (UniqueName: \"kubernetes.io/projected/37a5fadd-c499-4540-b603-1fb4b01821d1-kube-api-access-pxm2r\") pod \"37a5fadd-c499-4540-b603-1fb4b01821d1\" (UID: \"37a5fadd-c499-4540-b603-1fb4b01821d1\") " Oct 31 00:45:03.873024 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:03.872944 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37a5fadd-c499-4540-b603-1fb4b01821d1-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/37a5fadd-c499-4540-b603-1fb4b01821d1-otlp-tls-certs\") pod \"37a5fadd-c499-4540-b603-1fb4b01821d1\" (UID: \"37a5fadd-c499-4540-b603-1fb4b01821d1\") " Oct 31 00:45:03.873024 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:03.873017 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37a5fadd-c499-4540-b603-1fb4b01821d1-otc-internal\" (UniqueName: \"kubernetes.io/configmap/37a5fadd-c499-4540-b603-1fb4b01821d1-otc-internal\") pod \"37a5fadd-c499-4540-b603-1fb4b01821d1\" (UID: \"37a5fadd-c499-4540-b603-1fb4b01821d1\") " Oct 31 00:45:03.873407 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:03.873385 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37a5fadd-c499-4540-b603-1fb4b01821d1-otc-internal" pod "37a5fadd-c499-4540-b603-1fb4b01821d1" (UID: "37a5fadd-c499-4540-b603-1fb4b01821d1"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:03.874816 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:03.874796 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37a5fadd-c499-4540-b603-1fb4b01821d1-kube-api-access-pxm2r" pod "37a5fadd-c499-4540-b603-1fb4b01821d1" (UID: "37a5fadd-c499-4540-b603-1fb4b01821d1"). InnerVolumeSpecName "kube-api-access-pxm2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:03.874912 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:03.874869 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37a5fadd-c499-4540-b603-1fb4b01821d1-otlp-tls-certs" pod "37a5fadd-c499-4540-b603-1fb4b01821d1" (UID: "37a5fadd-c499-4540-b603-1fb4b01821d1"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:03.974200 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:03.974163 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a9115f6-051d-42da-a1b3-a59cf69d0b03-kube-api-access-899ww\" (UniqueName: \"kubernetes.io/projected/0a9115f6-051d-42da-a1b3-a59cf69d0b03-kube-api-access-899ww\") pod \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\" (UID: \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\") " Oct 31 00:45:03.974325 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:03.974219 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a9115f6-051d-42da-a1b3-a59cf69d0b03-rbac\" (UniqueName: \"kubernetes.io/configmap/0a9115f6-051d-42da-a1b3-a59cf69d0b03-rbac\") pod \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\" (UID: \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\") " Oct 31 00:45:03.974325 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:03.974248 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a9115f6-051d-42da-a1b3-a59cf69d0b03-tls-secret\" (UniqueName: \"kubernetes.io/projected/0a9115f6-051d-42da-a1b3-a59cf69d0b03-tls-secret\") pod \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\" (UID: \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\") " Oct 31 00:45:03.974325 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:03.974275 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a9115f6-051d-42da-a1b3-a59cf69d0b03-data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/0a9115f6-051d-42da-a1b3-a59cf69d0b03-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\" (UID: \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\") " Oct 31 00:45:03.974492 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:03.974345 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a9115f6-051d-42da-a1b3-a59cf69d0b03-lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/0a9115f6-051d-42da-a1b3-a59cf69d0b03-lokistack-gateway\") pod \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\" (UID: \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\") " Oct 31 00:45:03.974492 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:03.974381 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a9115f6-051d-42da-a1b3-a59cf69d0b03-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a9115f6-051d-42da-a1b3-a59cf69d0b03-data-science-usage-logs-lokistack-ca-bundle\") pod \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\" (UID: \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\") " Oct 31 00:45:03.974492 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:03.974414 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a9115f6-051d-42da-a1b3-a59cf69d0b03-tenants\" (UniqueName: \"kubernetes.io/secret/0a9115f6-051d-42da-a1b3-a59cf69d0b03-tenants\") pod \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\" (UID: \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\") " Oct 31 00:45:03.974492 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:03.974454 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a9115f6-051d-42da-a1b3-a59cf69d0b03-data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/0a9115f6-051d-42da-a1b3-a59cf69d0b03-data-science-usage-logs-lokistack-gateway-client-http\") pod \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\" (UID: \"0a9115f6-051d-42da-a1b3-a59cf69d0b03\") " Oct 31 00:45:03.974710 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:03.974695 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxm2r\" (UniqueName: \"kubernetes.io/projected/37a5fadd-c499-4540-b603-1fb4b01821d1-kube-api-access-pxm2r\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:03.974773 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:03.974718 5274 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/37a5fadd-c499-4540-b603-1fb4b01821d1-otlp-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:03.974773 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:03.974732 5274 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/37a5fadd-c499-4540-b603-1fb4b01821d1-otc-internal\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:03.975443 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:03.974837 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a9115f6-051d-42da-a1b3-a59cf69d0b03-data-science-usage-logs-lokistack-ca-bundle" pod "0a9115f6-051d-42da-a1b3-a59cf69d0b03" (UID: "0a9115f6-051d-42da-a1b3-a59cf69d0b03"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:03.976650 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:03.976628 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9115f6-051d-42da-a1b3-a59cf69d0b03-data-science-usage-logs-lokistack-gateway-ca-bundle" pod "0a9115f6-051d-42da-a1b3-a59cf69d0b03" (UID: "0a9115f6-051d-42da-a1b3-a59cf69d0b03"). InnerVolumeSpecName "data-science-usage-logs-lokistack-gateway-ca-bundle". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:03.976827 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:03.976803 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9115f6-051d-42da-a1b3-a59cf69d0b03-kube-api-access-899ww" pod "0a9115f6-051d-42da-a1b3-a59cf69d0b03" (UID: "0a9115f6-051d-42da-a1b3-a59cf69d0b03"). InnerVolumeSpecName "kube-api-access-899ww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:03.977426 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:03.977403 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a9115f6-051d-42da-a1b3-a59cf69d0b03-data-science-usage-logs-lokistack-gateway-client-http" pod "0a9115f6-051d-42da-a1b3-a59cf69d0b03" (UID: "0a9115f6-051d-42da-a1b3-a59cf69d0b03"). InnerVolumeSpecName "data-science-usage-logs-lokistack-gateway-client-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:03.977480 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:03.977403 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9115f6-051d-42da-a1b3-a59cf69d0b03-tls-secret" pod "0a9115f6-051d-42da-a1b3-a59cf69d0b03" (UID: "0a9115f6-051d-42da-a1b3-a59cf69d0b03"). InnerVolumeSpecName "tls-secret". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:03.980682 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:03.980659 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a9115f6-051d-42da-a1b3-a59cf69d0b03-rbac" pod "0a9115f6-051d-42da-a1b3-a59cf69d0b03" (UID: "0a9115f6-051d-42da-a1b3-a59cf69d0b03"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:03.980846 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:03.980712 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a9115f6-051d-42da-a1b3-a59cf69d0b03-lokistack-gateway" pod "0a9115f6-051d-42da-a1b3-a59cf69d0b03" (UID: "0a9115f6-051d-42da-a1b3-a59cf69d0b03"). InnerVolumeSpecName "lokistack-gateway". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:03.981618 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:03.981600 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a9115f6-051d-42da-a1b3-a59cf69d0b03-tenants" pod "0a9115f6-051d-42da-a1b3-a59cf69d0b03" (UID: "0a9115f6-051d-42da-a1b3-a59cf69d0b03"). InnerVolumeSpecName "tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:04.076064 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:04.076004 5274 reconciler_common.go:299] "Volume detached for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/0a9115f6-051d-42da-a1b3-a59cf69d0b03-lokistack-gateway\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:04.076064 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:04.076035 5274 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a9115f6-051d-42da-a1b3-a59cf69d0b03-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:04.076064 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:04.076046 5274 reconciler_common.go:299] "Volume detached for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/0a9115f6-051d-42da-a1b3-a59cf69d0b03-tenants\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:04.076064 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:04.076056 5274 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/0a9115f6-051d-42da-a1b3-a59cf69d0b03-data-science-usage-logs-lokistack-gateway-client-http\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:04.076064 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:04.076065 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-899ww\" (UniqueName: \"kubernetes.io/projected/0a9115f6-051d-42da-a1b3-a59cf69d0b03-kube-api-access-899ww\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:04.076064 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:04.076074 5274 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/0a9115f6-051d-42da-a1b3-a59cf69d0b03-rbac\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:04.076313 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:04.076082 5274 reconciler_common.go:299] "Volume detached for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/0a9115f6-051d-42da-a1b3-a59cf69d0b03-tls-secret\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:04.076313 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:04.076091 5274 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/0a9115f6-051d-42da-a1b3-a59cf69d0b03-data-science-usage-logs-lokistack-gateway-ca-bundle\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:04.502278 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:04.502242 5274 generic.go:356] "Generic (PLEG): container finished" podID="0a9115f6-051d-42da-a1b3-a59cf69d0b03" containerID="0902458c8fe7a673ada51d8ed2f67d19ab310f56824202b4bff5a2526caab800" exitCode=0 Oct 31 00:45:04.502482 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:04.502290 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" Oct 31 00:45:04.502482 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:04.502321 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" event={"ID":"0a9115f6-051d-42da-a1b3-a59cf69d0b03","Type":"ContainerDied","Data":"0902458c8fe7a673ada51d8ed2f67d19ab310f56824202b4bff5a2526caab800"} Oct 31 00:45:04.502482 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:04.502354 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql" event={"ID":"0a9115f6-051d-42da-a1b3-a59cf69d0b03","Type":"ContainerDied","Data":"eb57543f5587d4b16c643d278f8c67c311a471d375a9e1a4b6541db422752ce0"} Oct 31 00:45:04.502482 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:04.502377 5274 scope.go:122] "RemoveContainer" containerID="e8612cad31b72fed8ba75ce1e73f56f05069a38d845d3b06dfb851d51b0ab291" Oct 31 00:45:04.504093 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:04.504044 5274 generic.go:356] "Generic (PLEG): container finished" podID="37a5fadd-c499-4540-b603-1fb4b01821d1" containerID="60c55da15380287f4d1fa657d48582abf54fe0986498eb8966287e3e041dcf6b" exitCode=0 Oct 31 00:45:04.504093 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:04.504086 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-swvbn" Oct 31 00:45:04.504274 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:04.504109 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-swvbn" event={"ID":"37a5fadd-c499-4540-b603-1fb4b01821d1","Type":"ContainerDied","Data":"60c55da15380287f4d1fa657d48582abf54fe0986498eb8966287e3e041dcf6b"} Oct 31 00:45:04.504274 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:04.504185 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-swvbn" event={"ID":"37a5fadd-c499-4540-b603-1fb4b01821d1","Type":"ContainerDied","Data":"7d72bca99057ac9c1a822b0ea731cc491b767d5e03a910a0af3cd6f4a37070fa"} Oct 31 00:45:04.515012 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:04.514984 5274 scope.go:122] "RemoveContainer" containerID="0902458c8fe7a673ada51d8ed2f67d19ab310f56824202b4bff5a2526caab800" Oct 31 00:45:04.530607 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:04.530581 5274 scope.go:122] "RemoveContainer" containerID="e8612cad31b72fed8ba75ce1e73f56f05069a38d845d3b06dfb851d51b0ab291" Oct 31 00:45:04.530678 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:04.530652 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql"] Oct 31 00:45:04.530925 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:45:04.530901 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8612cad31b72fed8ba75ce1e73f56f05069a38d845d3b06dfb851d51b0ab291\": container with ID starting with e8612cad31b72fed8ba75ce1e73f56f05069a38d845d3b06dfb851d51b0ab291 not found: ID does not exist" containerID="e8612cad31b72fed8ba75ce1e73f56f05069a38d845d3b06dfb851d51b0ab291" Oct 31 00:45:04.531008 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:04.530935 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8612cad31b72fed8ba75ce1e73f56f05069a38d845d3b06dfb851d51b0ab291"} err="failed to get container status \"e8612cad31b72fed8ba75ce1e73f56f05069a38d845d3b06dfb851d51b0ab291\": rpc error: code = NotFound desc = could not find container \"e8612cad31b72fed8ba75ce1e73f56f05069a38d845d3b06dfb851d51b0ab291\": container with ID starting with e8612cad31b72fed8ba75ce1e73f56f05069a38d845d3b06dfb851d51b0ab291 not found: ID does not exist" Oct 31 00:45:04.531008 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:04.530998 5274 scope.go:122] "RemoveContainer" containerID="0902458c8fe7a673ada51d8ed2f67d19ab310f56824202b4bff5a2526caab800" Oct 31 00:45:04.531332 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:45:04.531304 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0902458c8fe7a673ada51d8ed2f67d19ab310f56824202b4bff5a2526caab800\": container with ID starting with 0902458c8fe7a673ada51d8ed2f67d19ab310f56824202b4bff5a2526caab800 not found: ID does not exist" containerID="0902458c8fe7a673ada51d8ed2f67d19ab310f56824202b4bff5a2526caab800" Oct 31 00:45:04.531413 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:04.531334 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0902458c8fe7a673ada51d8ed2f67d19ab310f56824202b4bff5a2526caab800"} err="failed to get container status \"0902458c8fe7a673ada51d8ed2f67d19ab310f56824202b4bff5a2526caab800\": rpc error: code = NotFound desc = could not find container \"0902458c8fe7a673ada51d8ed2f67d19ab310f56824202b4bff5a2526caab800\": container with ID starting with 0902458c8fe7a673ada51d8ed2f67d19ab310f56824202b4bff5a2526caab800 not found: ID does not exist" Oct 31 00:45:04.531413 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:04.531349 5274 scope.go:122] "RemoveContainer" containerID="60c55da15380287f4d1fa657d48582abf54fe0986498eb8966287e3e041dcf6b" Oct 31 00:45:04.535999 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:04.535976 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-5cb78dfd46-svlql"] Oct 31 00:45:04.540500 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:04.540482 5274 scope.go:122] "RemoveContainer" containerID="60c55da15380287f4d1fa657d48582abf54fe0986498eb8966287e3e041dcf6b" Oct 31 00:45:04.540818 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:45:04.540791 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60c55da15380287f4d1fa657d48582abf54fe0986498eb8966287e3e041dcf6b\": container with ID starting with 60c55da15380287f4d1fa657d48582abf54fe0986498eb8966287e3e041dcf6b not found: ID does not exist" containerID="60c55da15380287f4d1fa657d48582abf54fe0986498eb8966287e3e041dcf6b" Oct 31 00:45:04.540888 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:04.540831 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60c55da15380287f4d1fa657d48582abf54fe0986498eb8966287e3e041dcf6b"} err="failed to get container status \"60c55da15380287f4d1fa657d48582abf54fe0986498eb8966287e3e041dcf6b\": rpc error: code = NotFound desc = could not find container \"60c55da15380287f4d1fa657d48582abf54fe0986498eb8966287e3e041dcf6b\": container with ID starting with 60c55da15380287f4d1fa657d48582abf54fe0986498eb8966287e3e041dcf6b not found: ID does not exist" Oct 31 00:45:04.549776 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:04.549750 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-swvbn"] Oct 31 00:45:04.556047 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:04.556025 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-swvbn"] Oct 31 00:45:05.480494 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:05.480458 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a9115f6-051d-42da-a1b3-a59cf69d0b03" path="/var/lib/kubelet/pods/0a9115f6-051d-42da-a1b3-a59cf69d0b03/volumes" Oct 31 00:45:05.480892 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:05.480875 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37a5fadd-c499-4540-b603-1fb4b01821d1" path="/var/lib/kubelet/pods/37a5fadd-c499-4540-b603-1fb4b01821d1/volumes" Oct 31 00:45:20.899203 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:45:20.899168 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:45:37.379649 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.379610 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-mpkxh"] Oct 31 00:45:37.380094 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.379996 5274 cpu_manager.go:414] "containerMap: removing container" podUID="37a5fadd-c499-4540-b603-1fb4b01821d1" containerName="otc-container" Oct 31 00:45:37.380094 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.380009 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37a5fadd-c499-4540-b603-1fb4b01821d1" containerName="otc-container" Oct 31 00:45:37.380094 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.380033 5274 cpu_manager.go:414] "containerMap: removing container" podUID="0a9115f6-051d-42da-a1b3-a59cf69d0b03" containerName="gateway" Oct 31 00:45:37.380094 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.380039 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a9115f6-051d-42da-a1b3-a59cf69d0b03" containerName="gateway" Oct 31 00:45:37.380094 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.380046 5274 cpu_manager.go:414] "containerMap: removing container" podUID="0a9115f6-051d-42da-a1b3-a59cf69d0b03" containerName="opa" Oct 31 00:45:37.380094 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.380051 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a9115f6-051d-42da-a1b3-a59cf69d0b03" containerName="opa" Oct 31 00:45:37.380287 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.380113 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="37a5fadd-c499-4540-b603-1fb4b01821d1" containerName="otc-container" Oct 31 00:45:37.380287 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.380124 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a9115f6-051d-42da-a1b3-a59cf69d0b03" containerName="gateway" Oct 31 00:45:37.380287 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.380130 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a9115f6-051d-42da-a1b3-a59cf69d0b03" containerName="opa" Oct 31 00:45:37.384692 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.384673 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-mpkxh" Oct 31 00:45:37.394841 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.394819 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:45:37.395558 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.395476 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-6flx5\"" Oct 31 00:45:37.395558 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.395499 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:45:37.395558 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.395510 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 31 00:45:37.395558 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.395521 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:45:37.395558 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.395519 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\"" Oct 31 00:45:37.395558 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.395486 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\"" Oct 31 00:45:37.396836 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.396814 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-mpkxh"] Oct 31 00:45:37.439348 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.439323 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-n684d"] Oct 31 00:45:37.442800 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.442783 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-n684d" Oct 31 00:45:37.452404 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.452381 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-n684d"] Oct 31 00:45:37.475142 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.475108 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 31 00:45:37.475250 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.475187 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-2x7mp\"" Oct 31 00:45:37.496022 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.496006 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 31 00:45:37.537493 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.537470 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cdbf3648-1abd-4d7b-959d-637a1e564adf-config\") pod \"korrel8r-5cbcc89b8f-n684d\" (UID: \"cdbf3648-1abd-4d7b-959d-637a1e564adf\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-n684d" Oct 31 00:45:37.537596 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.537494 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/cdbf3648-1abd-4d7b-959d-637a1e564adf-tls\") pod \"korrel8r-5cbcc89b8f-n684d\" (UID: \"cdbf3648-1abd-4d7b-959d-637a1e564adf\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-n684d" Oct 31 00:45:37.537596 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.537515 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/644488ea-9480-4d33-9e58-701b46b95a13-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-mpkxh\" (UID: \"644488ea-9480-4d33-9e58-701b46b95a13\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-mpkxh" Oct 31 00:45:37.537596 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.537534 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/644488ea-9480-4d33-9e58-701b46b95a13-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-mpkxh\" (UID: \"644488ea-9480-4d33-9e58-701b46b95a13\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-mpkxh" Oct 31 00:45:37.537714 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.537697 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkzx6\" (UniqueName: \"kubernetes.io/projected/cdbf3648-1abd-4d7b-959d-637a1e564adf-kube-api-access-gkzx6\") pod \"korrel8r-5cbcc89b8f-n684d\" (UID: \"cdbf3648-1abd-4d7b-959d-637a1e564adf\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-n684d" Oct 31 00:45:37.537755 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.537730 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/644488ea-9480-4d33-9e58-701b46b95a13-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-mpkxh\" (UID: \"644488ea-9480-4d33-9e58-701b46b95a13\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-mpkxh" Oct 31 00:45:37.537755 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.537749 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66djb\" (UniqueName: \"kubernetes.io/projected/644488ea-9480-4d33-9e58-701b46b95a13-kube-api-access-66djb\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-mpkxh\" (UID: \"644488ea-9480-4d33-9e58-701b46b95a13\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-mpkxh" Oct 31 00:45:37.537848 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.537826 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/644488ea-9480-4d33-9e58-701b46b95a13-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-mpkxh\" (UID: \"644488ea-9480-4d33-9e58-701b46b95a13\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-mpkxh" Oct 31 00:45:37.638912 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.638838 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/cdbf3648-1abd-4d7b-959d-637a1e564adf-tls\") pod \"korrel8r-5cbcc89b8f-n684d\" (UID: \"cdbf3648-1abd-4d7b-959d-637a1e564adf\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-n684d" Oct 31 00:45:37.638912 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.638885 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/644488ea-9480-4d33-9e58-701b46b95a13-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-mpkxh\" (UID: \"644488ea-9480-4d33-9e58-701b46b95a13\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-mpkxh" Oct 31 00:45:37.639128 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.638932 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/644488ea-9480-4d33-9e58-701b46b95a13-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-mpkxh\" (UID: \"644488ea-9480-4d33-9e58-701b46b95a13\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-mpkxh" Oct 31 00:45:37.639128 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.639051 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkzx6\" (UniqueName: \"kubernetes.io/projected/cdbf3648-1abd-4d7b-959d-637a1e564adf-kube-api-access-gkzx6\") pod \"korrel8r-5cbcc89b8f-n684d\" (UID: \"cdbf3648-1abd-4d7b-959d-637a1e564adf\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-n684d" Oct 31 00:45:37.639128 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.639085 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/644488ea-9480-4d33-9e58-701b46b95a13-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-mpkxh\" (UID: \"644488ea-9480-4d33-9e58-701b46b95a13\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-mpkxh" Oct 31 00:45:37.640270 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.640248 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/644488ea-9480-4d33-9e58-701b46b95a13-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-mpkxh\" (UID: \"644488ea-9480-4d33-9e58-701b46b95a13\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-mpkxh" Oct 31 00:45:37.640401 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.640297 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66djb\" (UniqueName: \"kubernetes.io/projected/644488ea-9480-4d33-9e58-701b46b95a13-kube-api-access-66djb\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-mpkxh\" (UID: \"644488ea-9480-4d33-9e58-701b46b95a13\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-mpkxh" Oct 31 00:45:37.640401 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.640339 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/644488ea-9480-4d33-9e58-701b46b95a13-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-mpkxh\" (UID: \"644488ea-9480-4d33-9e58-701b46b95a13\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-mpkxh" Oct 31 00:45:37.640401 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.640382 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cdbf3648-1abd-4d7b-959d-637a1e564adf-config\") pod \"korrel8r-5cbcc89b8f-n684d\" (UID: \"cdbf3648-1abd-4d7b-959d-637a1e564adf\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-n684d" Oct 31 00:45:37.641366 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.641344 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cdbf3648-1abd-4d7b-959d-637a1e564adf-config\") pod \"korrel8r-5cbcc89b8f-n684d\" (UID: \"cdbf3648-1abd-4d7b-959d-637a1e564adf\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-n684d" Oct 31 00:45:37.641594 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:45:37.641577 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found Oct 31 00:45:37.641745 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:45:37.641733 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/644488ea-9480-4d33-9e58-701b46b95a13-prometheus-client-cert podName:644488ea-9480-4d33-9e58-701b46b95a13 nodeName:}" failed. No retries permitted until 2026-10-31 00:45:38.141712223 +0000 UTC m=+2774.778530934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/644488ea-9480-4d33-9e58-701b46b95a13-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-mpkxh" (UID: "644488ea-9480-4d33-9e58-701b46b95a13") : secret "prometheus-operated-tls" not found Oct 31 00:45:37.642572 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.642549 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/644488ea-9480-4d33-9e58-701b46b95a13-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-mpkxh\" (UID: \"644488ea-9480-4d33-9e58-701b46b95a13\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-mpkxh" Oct 31 00:45:37.643802 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.643779 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/644488ea-9480-4d33-9e58-701b46b95a13-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-mpkxh\" (UID: \"644488ea-9480-4d33-9e58-701b46b95a13\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-mpkxh" Oct 31 00:45:37.643986 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.643909 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/cdbf3648-1abd-4d7b-959d-637a1e564adf-tls\") pod \"korrel8r-5cbcc89b8f-n684d\" (UID: \"cdbf3648-1abd-4d7b-959d-637a1e564adf\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-n684d" Oct 31 00:45:37.658761 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.658736 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66djb\" (UniqueName: \"kubernetes.io/projected/644488ea-9480-4d33-9e58-701b46b95a13-kube-api-access-66djb\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-mpkxh\" (UID: \"644488ea-9480-4d33-9e58-701b46b95a13\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-mpkxh" Oct 31 00:45:37.660001 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.659944 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkzx6\" (UniqueName: \"kubernetes.io/projected/cdbf3648-1abd-4d7b-959d-637a1e564adf-kube-api-access-gkzx6\") pod \"korrel8r-5cbcc89b8f-n684d\" (UID: \"cdbf3648-1abd-4d7b-959d-637a1e564adf\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-n684d" Oct 31 00:45:37.755834 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.755809 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-n684d" Oct 31 00:45:37.919609 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.919519 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:45:37.924121 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.924097 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:45:37.930676 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.930261 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:45:37.930676 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.930391 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:45:37.930676 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.930439 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:45:37.930676 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.930541 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:45:37.930676 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.930644 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:45:37.934989 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:37.934942 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:45:38.040911 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:38.039328 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-n684d"] Oct 31 00:45:38.042320 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:45:38.042271 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdbf3648_1abd_4d7b_959d_637a1e564adf.slice/crio-ace4ca5dced3d0227bbf1f63a7e549830dda43e05a6e02952b927d94fcac4aef WatchSource:0}: Error finding container ace4ca5dced3d0227bbf1f63a7e549830dda43e05a6e02952b927d94fcac4aef: Status 404 returned error can't find the container with id ace4ca5dced3d0227bbf1f63a7e549830dda43e05a6e02952b927d94fcac4aef Oct 31 00:45:38.043400 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:38.043311 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/07f5e099-eae5-418f-864c-1b1fd079b595-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"07f5e099-eae5-418f-864c-1b1fd079b595\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:45:38.043400 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:38.043373 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szlb5\" (UniqueName: \"kubernetes.io/projected/07f5e099-eae5-418f-864c-1b1fd079b595-kube-api-access-szlb5\") pod \"data-science-collector-collector-0\" (UID: \"07f5e099-eae5-418f-864c-1b1fd079b595\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:45:38.043528 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:38.043427 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/07f5e099-eae5-418f-864c-1b1fd079b595-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"07f5e099-eae5-418f-864c-1b1fd079b595\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:45:38.043528 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:38.043446 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07f5e099-eae5-418f-864c-1b1fd079b595-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"07f5e099-eae5-418f-864c-1b1fd079b595\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:45:38.043528 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:38.043487 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/07f5e099-eae5-418f-864c-1b1fd079b595-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"07f5e099-eae5-418f-864c-1b1fd079b595\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:45:38.043528 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:38.043509 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/07f5e099-eae5-418f-864c-1b1fd079b595-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"07f5e099-eae5-418f-864c-1b1fd079b595\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:45:38.144508 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:38.144477 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/07f5e099-eae5-418f-864c-1b1fd079b595-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"07f5e099-eae5-418f-864c-1b1fd079b595\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:45:38.144646 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:38.144533 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szlb5\" (UniqueName: \"kubernetes.io/projected/07f5e099-eae5-418f-864c-1b1fd079b595-kube-api-access-szlb5\") pod \"data-science-collector-collector-0\" (UID: \"07f5e099-eae5-418f-864c-1b1fd079b595\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:45:38.144646 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:38.144602 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/07f5e099-eae5-418f-864c-1b1fd079b595-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"07f5e099-eae5-418f-864c-1b1fd079b595\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:45:38.144646 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:38.144629 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07f5e099-eae5-418f-864c-1b1fd079b595-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"07f5e099-eae5-418f-864c-1b1fd079b595\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:45:38.144812 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:38.144664 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/644488ea-9480-4d33-9e58-701b46b95a13-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-mpkxh\" (UID: \"644488ea-9480-4d33-9e58-701b46b95a13\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-mpkxh" Oct 31 00:45:38.144812 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:38.144695 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/07f5e099-eae5-418f-864c-1b1fd079b595-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"07f5e099-eae5-418f-864c-1b1fd079b595\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:45:38.144812 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:38.144734 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/07f5e099-eae5-418f-864c-1b1fd079b595-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"07f5e099-eae5-418f-864c-1b1fd079b595\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:45:38.144812 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:45:38.144738 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found Oct 31 00:45:38.144812 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:45:38.144811 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07f5e099-eae5-418f-864c-1b1fd079b595-otlp-tls-certs podName:07f5e099-eae5-418f-864c-1b1fd079b595 nodeName:}" failed. No retries permitted until 2026-10-31 00:45:38.64479222 +0000 UTC m=+2775.281610932 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/07f5e099-eae5-418f-864c-1b1fd079b595-otlp-tls-certs") pod "data-science-collector-collector-0" (UID: "07f5e099-eae5-418f-864c-1b1fd079b595") : secret "data-science-collector-collector-headless-tls" not found Oct 31 00:45:38.145458 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:38.145436 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/07f5e099-eae5-418f-864c-1b1fd079b595-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"07f5e099-eae5-418f-864c-1b1fd079b595\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:45:38.147111 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:38.147085 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/644488ea-9480-4d33-9e58-701b46b95a13-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-mpkxh\" (UID: \"644488ea-9480-4d33-9e58-701b46b95a13\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-mpkxh" Oct 31 00:45:38.147111 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:38.147108 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/07f5e099-eae5-418f-864c-1b1fd079b595-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"07f5e099-eae5-418f-864c-1b1fd079b595\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:45:38.147261 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:38.147127 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07f5e099-eae5-418f-864c-1b1fd079b595-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"07f5e099-eae5-418f-864c-1b1fd079b595\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:45:38.147444 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:38.147426 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/07f5e099-eae5-418f-864c-1b1fd079b595-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"07f5e099-eae5-418f-864c-1b1fd079b595\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:45:38.156664 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:38.156641 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szlb5\" (UniqueName: \"kubernetes.io/projected/07f5e099-eae5-418f-864c-1b1fd079b595-kube-api-access-szlb5\") pod \"data-science-collector-collector-0\" (UID: \"07f5e099-eae5-418f-864c-1b1fd079b595\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:45:38.298899 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:38.298825 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-mpkxh" Oct 31 00:45:38.571982 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:38.571924 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-mpkxh"] Oct 31 00:45:38.574439 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:45:38.574410 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod644488ea_9480_4d33_9e58_701b46b95a13.slice/crio-eaf6a0d0182765b7e3fa1780d42daeeb82f76a7205d1f78b3e21a0681c35dd5b WatchSource:0}: Error finding container eaf6a0d0182765b7e3fa1780d42daeeb82f76a7205d1f78b3e21a0681c35dd5b: Status 404 returned error can't find the container with id eaf6a0d0182765b7e3fa1780d42daeeb82f76a7205d1f78b3e21a0681c35dd5b Oct 31 00:45:38.649939 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:38.649909 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/07f5e099-eae5-418f-864c-1b1fd079b595-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"07f5e099-eae5-418f-864c-1b1fd079b595\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:45:38.650090 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:45:38.650058 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found Oct 31 00:45:38.650160 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:45:38.650122 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07f5e099-eae5-418f-864c-1b1fd079b595-otlp-tls-certs podName:07f5e099-eae5-418f-864c-1b1fd079b595 nodeName:}" failed. No retries permitted until 2026-10-31 00:45:39.650102893 +0000 UTC m=+2776.286921604 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/07f5e099-eae5-418f-864c-1b1fd079b595-otlp-tls-certs") pod "data-science-collector-collector-0" (UID: "07f5e099-eae5-418f-864c-1b1fd079b595") : secret "data-science-collector-collector-headless-tls" not found Oct 31 00:45:38.660901 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:38.660876 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-n684d" event={"ID":"cdbf3648-1abd-4d7b-959d-637a1e564adf","Type":"ContainerStarted","Data":"54f456929e1f583631a2e24f97d945cbc62ca6471233d20880b808022bc19e30"} Oct 31 00:45:38.661053 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:38.660931 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-n684d" event={"ID":"cdbf3648-1abd-4d7b-959d-637a1e564adf","Type":"ContainerStarted","Data":"ace4ca5dced3d0227bbf1f63a7e549830dda43e05a6e02952b927d94fcac4aef"} Oct 31 00:45:38.661119 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:38.661072 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-n684d" Oct 31 00:45:38.662110 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:38.662091 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-mpkxh" event={"ID":"644488ea-9480-4d33-9e58-701b46b95a13","Type":"ContainerStarted","Data":"eaf6a0d0182765b7e3fa1780d42daeeb82f76a7205d1f78b3e21a0681c35dd5b"} Oct 31 00:45:38.681869 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:38.681828 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-n684d" podStartSLOduration=1.681816321 podStartE2EDuration="1.681816321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:38.681191501 +0000 UTC m=+2775.318010219" watchObservedRunningTime="2026-10-31 00:45:38.681816321 +0000 UTC m=+2775.318635066" Oct 31 00:45:39.656388 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:39.656359 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/07f5e099-eae5-418f-864c-1b1fd079b595-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"07f5e099-eae5-418f-864c-1b1fd079b595\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:45:39.658758 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:39.658737 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/07f5e099-eae5-418f-864c-1b1fd079b595-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"07f5e099-eae5-418f-864c-1b1fd079b595\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:45:39.669242 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:39.669207 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-mpkxh" event={"ID":"644488ea-9480-4d33-9e58-701b46b95a13","Type":"ContainerStarted","Data":"23a18d03209ee6ee91728c4e12eb7e7224323ee51b16c3955a3ef61fad2a27a9"} Oct 31 00:45:39.669368 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:39.669349 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-mpkxh" Oct 31 00:45:39.691108 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:39.691062 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-mpkxh" podStartSLOduration=2.691049408 podStartE2EDuration="2.691049408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:39.688856743 +0000 UTC m=+2776.325675460" watchObservedRunningTime="2026-10-31 00:45:39.691049408 +0000 UTC m=+2776.327868123" Oct 31 00:45:39.742635 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:39.742615 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:45:40.019203 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:40.019136 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:45:40.022270 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:45:40.022243 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f5e099_eae5_418f_864c_1b1fd079b595.slice/crio-1c1939a687ce24c9d2d4584b6c3678066df474d035f661381797d84c0a146fa2 WatchSource:0}: Error finding container 1c1939a687ce24c9d2d4584b6c3678066df474d035f661381797d84c0a146fa2: Status 404 returned error can't find the container with id 1c1939a687ce24c9d2d4584b6c3678066df474d035f661381797d84c0a146fa2 Oct 31 00:45:40.024443 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:40.024404 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:45:40.024533 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:40.024481 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:45:40.676747 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:40.676706 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"07f5e099-eae5-418f-864c-1b1fd079b595","Type":"ContainerStarted","Data":"5d8beb64989135944f02e20ba5a3bec199cc89ff79a03d3bf13ca5dc804b46c3"} Oct 31 00:45:40.677202 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:40.676796 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"07f5e099-eae5-418f-864c-1b1fd079b595","Type":"ContainerStarted","Data":"1c1939a687ce24c9d2d4584b6c3678066df474d035f661381797d84c0a146fa2"} Oct 31 00:45:40.700122 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:40.700081 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=3.700068919 podStartE2EDuration="3.700068919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:40.698482888 +0000 UTC m=+2777.335301606" watchObservedRunningTime="2026-10-31 00:45:40.700068919 +0000 UTC m=+2777.336887698" Oct 31 00:45:44.272883 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.272833 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:45:44.276791 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.276768 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:45:44.290788 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.290760 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:45:44.294896 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.294875 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3f69d683-62af-468f-8ae5-f2b2d101bf61-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3f69d683-62af-468f-8ae5-f2b2d101bf61\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:45:44.295105 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.294995 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3f69d683-62af-468f-8ae5-f2b2d101bf61-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3f69d683-62af-468f-8ae5-f2b2d101bf61\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:45:44.295105 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.295073 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/3f69d683-62af-468f-8ae5-f2b2d101bf61-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3f69d683-62af-468f-8ae5-f2b2d101bf61\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:45:44.295228 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.295110 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3f69d683-62af-468f-8ae5-f2b2d101bf61-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3f69d683-62af-468f-8ae5-f2b2d101bf61\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:45:44.295228 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.295154 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r725b\" (UniqueName: \"kubernetes.io/projected/3f69d683-62af-468f-8ae5-f2b2d101bf61-kube-api-access-r725b\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3f69d683-62af-468f-8ae5-f2b2d101bf61\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:45:44.295228 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.295205 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3f69d683-62af-468f-8ae5-f2b2d101bf61-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3f69d683-62af-468f-8ae5-f2b2d101bf61\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:45:44.295389 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.295229 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3f69d683-62af-468f-8ae5-f2b2d101bf61-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3f69d683-62af-468f-8ae5-f2b2d101bf61\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:45:44.295931 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.295775 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:45:44.300043 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.300026 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:44.314229 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.314162 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:45:44.371832 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.371801 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 31 00:45:44.371832 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.371833 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:45:44.372099 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.371883 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 31 00:45:44.372099 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.371896 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 31 00:45:44.372099 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.371909 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-gdvlc\"" Oct 31 00:45:44.372099 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.372013 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:45:44.372099 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.372053 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 31 00:45:44.372311 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.372108 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:45:44.372311 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.372140 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 31 00:45:44.372311 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.372180 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 31 00:45:44.372311 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.372185 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-b6hd7\"" Oct 31 00:45:44.385003 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.384950 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 31 00:45:44.385003 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.384982 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 31 00:45:44.385132 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.385084 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 31 00:45:44.396029 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.396006 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:44.396129 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.396032 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:44.396129 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.396062 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3f69d683-62af-468f-8ae5-f2b2d101bf61-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3f69d683-62af-468f-8ae5-f2b2d101bf61\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:45:44.396259 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.396196 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:44.396259 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.396238 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3f69d683-62af-468f-8ae5-f2b2d101bf61-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3f69d683-62af-468f-8ae5-f2b2d101bf61\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:45:44.396331 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.396282 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:44.396331 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.396308 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:44.396331 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.396327 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:44.396482 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.396344 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wnjg\" (UniqueName: \"kubernetes.io/projected/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-kube-api-access-6wnjg\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:44.396482 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.396459 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/3f69d683-62af-468f-8ae5-f2b2d101bf61-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3f69d683-62af-468f-8ae5-f2b2d101bf61\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:45:44.396586 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.396492 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:44.396586 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.396534 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3f69d683-62af-468f-8ae5-f2b2d101bf61-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3f69d683-62af-468f-8ae5-f2b2d101bf61\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:45:44.396676 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.396593 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:44.396676 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.396627 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:44.396793 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.396707 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:44.396793 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.396781 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r725b\" (UniqueName: \"kubernetes.io/projected/3f69d683-62af-468f-8ae5-f2b2d101bf61-kube-api-access-r725b\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3f69d683-62af-468f-8ae5-f2b2d101bf61\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:45:44.396902 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.396803 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/3f69d683-62af-468f-8ae5-f2b2d101bf61-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3f69d683-62af-468f-8ae5-f2b2d101bf61\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:45:44.396902 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.396815 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:44.396902 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.396866 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3f69d683-62af-468f-8ae5-f2b2d101bf61-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3f69d683-62af-468f-8ae5-f2b2d101bf61\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:45:44.397090 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.396911 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3f69d683-62af-468f-8ae5-f2b2d101bf61-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3f69d683-62af-468f-8ae5-f2b2d101bf61\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:45:44.398688 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.398667 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3f69d683-62af-468f-8ae5-f2b2d101bf61-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3f69d683-62af-468f-8ae5-f2b2d101bf61\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:45:44.399208 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.399182 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3f69d683-62af-468f-8ae5-f2b2d101bf61-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3f69d683-62af-468f-8ae5-f2b2d101bf61\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:45:44.399294 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.399192 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3f69d683-62af-468f-8ae5-f2b2d101bf61-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3f69d683-62af-468f-8ae5-f2b2d101bf61\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:45:44.399294 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.399248 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3f69d683-62af-468f-8ae5-f2b2d101bf61-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3f69d683-62af-468f-8ae5-f2b2d101bf61\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:45:44.399294 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.399288 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3f69d683-62af-468f-8ae5-f2b2d101bf61-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3f69d683-62af-468f-8ae5-f2b2d101bf61\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:45:44.408844 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.408823 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r725b\" (UniqueName: \"kubernetes.io/projected/3f69d683-62af-468f-8ae5-f2b2d101bf61-kube-api-access-r725b\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"3f69d683-62af-468f-8ae5-f2b2d101bf61\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:45:44.497583 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.497558 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:44.497692 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.497587 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:44.497692 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.497624 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:44.497692 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.497656 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:44.497692 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.497681 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:44.497917 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.497709 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:44.497917 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.497734 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wnjg\" (UniqueName: \"kubernetes.io/projected/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-kube-api-access-6wnjg\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:44.497917 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.497771 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:44.497917 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.497813 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:44.497917 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.497836 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:44.498661 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.498631 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:44.498781 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.498695 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:44.498781 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.498749 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:44.499493 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.499464 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:44.500033 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.500010 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:44.500448 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.500421 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:44.500531 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.500507 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:44.500636 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.500613 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:44.500805 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.500772 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:44.501115 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.501096 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:44.501719 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.501694 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:44.502094 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.502072 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:44.502213 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.502194 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:44.512696 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.512678 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wnjg\" (UniqueName: \"kubernetes.io/projected/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-kube-api-access-6wnjg\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:44.595306 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.595259 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:45:44.876864 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:44.876838 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:45:44.878468 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:45:44.878444 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f69d683_62af_468f_8ae5_f2b2d101bf61.slice/crio-46d1d1d231eb87eb82b0e555839dff04aadb02d9f8bf766686f9c52642f4dd8c WatchSource:0}: Error finding container 46d1d1d231eb87eb82b0e555839dff04aadb02d9f8bf766686f9c52642f4dd8c: Status 404 returned error can't find the container with id 46d1d1d231eb87eb82b0e555839dff04aadb02d9f8bf766686f9c52642f4dd8c Oct 31 00:45:45.679639 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:45.679613 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-mpkxh" Oct 31 00:45:45.702161 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:45.702132 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"3f69d683-62af-468f-8ae5-f2b2d101bf61","Type":"ContainerStarted","Data":"003e2f64063208af10073e83c74055a6ec87b08bdcf579f15067dc9a92078124"} Oct 31 00:45:45.702284 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:45.702179 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"3f69d683-62af-468f-8ae5-f2b2d101bf61","Type":"ContainerStarted","Data":"46d1d1d231eb87eb82b0e555839dff04aadb02d9f8bf766686f9c52642f4dd8c"} Oct 31 00:45:48.720222 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:48.720189 5274 generic.go:356] "Generic (PLEG): container finished" podID="3f69d683-62af-468f-8ae5-f2b2d101bf61" containerID="003e2f64063208af10073e83c74055a6ec87b08bdcf579f15067dc9a92078124" exitCode=0 Oct 31 00:45:48.720687 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:48.720233 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"3f69d683-62af-468f-8ae5-f2b2d101bf61","Type":"ContainerDied","Data":"003e2f64063208af10073e83c74055a6ec87b08bdcf579f15067dc9a92078124"} Oct 31 00:45:49.671636 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:49.671608 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-n684d" Oct 31 00:45:49.727758 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:49.727727 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"3f69d683-62af-468f-8ae5-f2b2d101bf61","Type":"ContainerStarted","Data":"eb445f800eb5fda3011cc89baf7988dce7a671257953e71fa9453edb67b6b314"} Oct 31 00:45:49.728138 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:49.727814 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"3f69d683-62af-468f-8ae5-f2b2d101bf61","Type":"ContainerStarted","Data":"2273a440ff8b094cfffc936bf77205754ee60a3b4410bfc11ee6d365776f13ab"} Oct 31 00:45:49.728138 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:49.728019 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:45:49.754410 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:49.754356 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=2.568148014 podStartE2EDuration="5.754340421s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.186192407s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:49.752990575 +0000 UTC m=+2786.389809291" watchObservedRunningTime="2026-10-31 00:45:49.754340421 +0000 UTC m=+2786.391159140" Oct 31 00:45:49.838642 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:49.838612 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:49.845906 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:49.845881 5274 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:49.940026 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:49.939950 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:49.940109 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:49.940082 5274 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:49.944795 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:49.944779 5274 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") DevicePath \"csi-3beb91b3ebf177571ca474589da3560cd18787cfa50b2a0b9bfef85661e6cf21\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:50.035661 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:50.035638 5274 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/534eb232558c8505fa55558aa87cc96e2d2b0adb946edb717939ec387be54e97/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:50.051673 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:50.051649 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:50.320644 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:50.320581 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:45:50.604186 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:50.604158 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:45:50.605941 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:45:50.605915 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode56adeab_d6b1_4e66_8732_8af1bd20ebd5.slice/crio-bbf42d9f709388a7850753a73a5f37a9edf59a472dfbeb00303bd08f7b75ca4c WatchSource:0}: Error finding container bbf42d9f709388a7850753a73a5f37a9edf59a472dfbeb00303bd08f7b75ca4c: Status 404 returned error can't find the container with id bbf42d9f709388a7850753a73a5f37a9edf59a472dfbeb00303bd08f7b75ca4c Oct 31 00:45:50.734148 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:50.734118 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"e56adeab-d6b1-4e66-8732-8af1bd20ebd5","Type":"ContainerStarted","Data":"bbf42d9f709388a7850753a73a5f37a9edf59a472dfbeb00303bd08f7b75ca4c"} Oct 31 00:45:50.899814 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:45:50.899775 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:45:51.740557 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:51.740525 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"e56adeab-d6b1-4e66-8732-8af1bd20ebd5","Type":"ContainerStarted","Data":"216d1ab841101a1cf1d7a33d1876ae0da0b95969b8b068c69a459fafc42177d7"} Oct 31 00:45:54.761844 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:54.761811 5274 generic.go:356] "Generic (PLEG): container finished" podID="e56adeab-d6b1-4e66-8732-8af1bd20ebd5" containerID="216d1ab841101a1cf1d7a33d1876ae0da0b95969b8b068c69a459fafc42177d7" exitCode=0 Oct 31 00:45:54.762263 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:54.761857 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"e56adeab-d6b1-4e66-8732-8af1bd20ebd5","Type":"ContainerDied","Data":"216d1ab841101a1cf1d7a33d1876ae0da0b95969b8b068c69a459fafc42177d7"} Oct 31 00:45:55.735560 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:55.735530 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:45:55.770180 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:55.770146 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"e56adeab-d6b1-4e66-8732-8af1bd20ebd5","Type":"ContainerStarted","Data":"0ed5b339f769a73202175ced3d09a3215b4ac05314e9f28fa15b9ea6ad3e960a"} Oct 31 00:45:55.770622 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:55.770233 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"e56adeab-d6b1-4e66-8732-8af1bd20ebd5","Type":"ContainerStarted","Data":"f633dbab80c0a149980bba3f0671096e25eacdbbc09c0ab613b947ac7d6e0767"} Oct 31 00:45:55.770622 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:55.770263 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"e56adeab-d6b1-4e66-8732-8af1bd20ebd5","Type":"ContainerStarted","Data":"47515dc25ec84115f53712b6b827624f0fac31e4801faf76867a74573df72162"} Oct 31 00:45:55.803173 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:45:55.803130 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=8.517871281 podStartE2EDuration="11.803116205s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.285244924s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:55.801048799 +0000 UTC m=+2792.437867516" watchObservedRunningTime="2026-10-31 00:45:55.803116205 +0000 UTC m=+2792.439934964" Oct 31 00:46:00.321562 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:00.321532 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:05.320769 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:05.320736 5274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:05.326181 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:05.326160 5274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:05.832873 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:05.832843 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:18.801279 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:18.801247 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:46:18.801726 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:18.801484 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="e56adeab-d6b1-4e66-8732-8af1bd20ebd5" containerName="prometheus" containerID="cri-o://47515dc25ec84115f53712b6b827624f0fac31e4801faf76867a74573df72162" gracePeriod=600 Oct 31 00:46:18.801726 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:18.801532 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="e56adeab-d6b1-4e66-8732-8af1bd20ebd5" containerName="thanos-sidecar" containerID="cri-o://0ed5b339f769a73202175ced3d09a3215b4ac05314e9f28fa15b9ea6ad3e960a" gracePeriod=600 Oct 31 00:46:18.801726 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:18.801548 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="e56adeab-d6b1-4e66-8732-8af1bd20ebd5" containerName="config-reloader" containerID="cri-o://f633dbab80c0a149980bba3f0671096e25eacdbbc09c0ab613b947ac7d6e0767" gracePeriod=600 Oct 31 00:46:19.545446 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.545420 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:19.679163 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.679093 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-config-out\" (UniqueName: \"kubernetes.io/empty-dir/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-config-out\") pod \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " Oct 31 00:46:19.679163 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.679133 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " Oct 31 00:46:19.679163 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.679163 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " Oct 31 00:46:19.679459 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.679190 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-tls-assets\" (UniqueName: \"kubernetes.io/projected/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-tls-assets\") pod \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " Oct 31 00:46:19.679459 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.679230 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-web-config\" (UniqueName: \"kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-web-config\") pod \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " Oct 31 00:46:19.679459 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.679335 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " Oct 31 00:46:19.679459 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.679374 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-secret-prometheus-web-tls-ca\") pod \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " Oct 31 00:46:19.679459 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.679421 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " Oct 31 00:46:19.679459 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.679450 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-config\" (UniqueName: \"kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-config\") pod \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " Oct 31 00:46:19.679777 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.679499 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-thanos-prometheus-http-client-file\") pod \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " Oct 31 00:46:19.679777 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.679532 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-kube-api-access-6wnjg\" (UniqueName: \"kubernetes.io/projected/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-kube-api-access-6wnjg\") pod \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " Oct 31 00:46:19.679777 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.679531 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-prometheus-data-science-monitoringstack-rulefiles-1" pod "e56adeab-d6b1-4e66-8732-8af1bd20ebd5" (UID: "e56adeab-d6b1-4e66-8732-8af1bd20ebd5"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:46:19.679777 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.679595 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " Oct 31 00:46:19.679777 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.679659 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\" (UID: \"e56adeab-d6b1-4e66-8732-8af1bd20ebd5\") " Oct 31 00:46:19.679777 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.679659 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-prometheus-data-science-monitoringstack-rulefiles-0" pod "e56adeab-d6b1-4e66-8732-8af1bd20ebd5" (UID: "e56adeab-d6b1-4e66-8732-8af1bd20ebd5"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:46:19.680245 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.680091 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-prometheus-data-science-monitoringstack-rulefiles-2" pod "e56adeab-d6b1-4e66-8732-8af1bd20ebd5" (UID: "e56adeab-d6b1-4e66-8732-8af1bd20ebd5"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:46:19.680245 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.680124 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:19.680245 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.680150 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:19.682466 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.682431 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-config-out" pod "e56adeab-d6b1-4e66-8732-8af1bd20ebd5" (UID: "e56adeab-d6b1-4e66-8732-8af1bd20ebd5"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:46:19.683161 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.683135 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-thanos-prometheus-http-client-file" pod "e56adeab-d6b1-4e66-8732-8af1bd20ebd5" (UID: "e56adeab-d6b1-4e66-8732-8af1bd20ebd5"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:19.683161 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.683151 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-config" pod "e56adeab-d6b1-4e66-8732-8af1bd20ebd5" (UID: "e56adeab-d6b1-4e66-8732-8af1bd20ebd5"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:19.683310 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.683198 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "e56adeab-d6b1-4e66-8732-8af1bd20ebd5" (UID: "e56adeab-d6b1-4e66-8732-8af1bd20ebd5"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:19.683310 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.683269 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-secret-prometheus-web-tls-ca" pod "e56adeab-d6b1-4e66-8732-8af1bd20ebd5" (UID: "e56adeab-d6b1-4e66-8732-8af1bd20ebd5"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:19.683698 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.683677 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "e56adeab-d6b1-4e66-8732-8af1bd20ebd5" (UID: "e56adeab-d6b1-4e66-8732-8af1bd20ebd5"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:19.683807 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.683721 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-tls-assets" pod "e56adeab-d6b1-4e66-8732-8af1bd20ebd5" (UID: "e56adeab-d6b1-4e66-8732-8af1bd20ebd5"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:46:19.683883 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.683864 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-kube-api-access-6wnjg" pod "e56adeab-d6b1-4e66-8732-8af1bd20ebd5" (UID: "e56adeab-d6b1-4e66-8732-8af1bd20ebd5"). InnerVolumeSpecName "kube-api-access-6wnjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:46:19.686257 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.686234 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a" pod "e56adeab-d6b1-4e66-8732-8af1bd20ebd5" (UID: "e56adeab-d6b1-4e66-8732-8af1bd20ebd5"). InnerVolumeSpecName "pvc-63e34709-8d85-4142-b53a-f655698606ff". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:46:19.697272 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.697245 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-web-config" pod "e56adeab-d6b1-4e66-8732-8af1bd20ebd5" (UID: "e56adeab-d6b1-4e66-8732-8af1bd20ebd5"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:19.780673 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.780647 5274 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-config-out\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:19.780673 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.780672 5274 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-tls-assets\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:19.780822 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.780682 5274 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-web-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:19.780822 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.780710 5274 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" " Oct 31 00:46:19.780822 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.780720 5274 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-secret-prometheus-web-tls-ca\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:19.780822 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.780730 5274 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:19.780822 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.780739 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:19.780822 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.780748 5274 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:19.780822 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.780756 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wnjg\" (UniqueName: \"kubernetes.io/projected/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-kube-api-access-6wnjg\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:19.780822 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.780766 5274 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:19.780822 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.780776 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e56adeab-d6b1-4e66-8732-8af1bd20ebd5-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:19.792596 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.792578 5274 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-63e34709-8d85-4142-b53a-f655698606ff" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a") on node "ip-10-0-129-70.us-east-2.compute.internal" Oct 31 00:46:19.881377 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.881355 5274 reconciler_common.go:299] "Volume detached for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"csi-3beb91b3ebf177571ca474589da3560cd18787cfa50b2a0b9bfef85661e6cf21\"" Oct 31 00:46:19.894278 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.894249 5274 generic.go:356] "Generic (PLEG): container finished" podID="e56adeab-d6b1-4e66-8732-8af1bd20ebd5" containerID="0ed5b339f769a73202175ced3d09a3215b4ac05314e9f28fa15b9ea6ad3e960a" exitCode=0 Oct 31 00:46:19.894278 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.894273 5274 generic.go:356] "Generic (PLEG): container finished" podID="e56adeab-d6b1-4e66-8732-8af1bd20ebd5" containerID="f633dbab80c0a149980bba3f0671096e25eacdbbc09c0ab613b947ac7d6e0767" exitCode=0 Oct 31 00:46:19.894278 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.894279 5274 generic.go:356] "Generic (PLEG): container finished" podID="e56adeab-d6b1-4e66-8732-8af1bd20ebd5" containerID="47515dc25ec84115f53712b6b827624f0fac31e4801faf76867a74573df72162" exitCode=0 Oct 31 00:46:19.894512 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.894314 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:19.894512 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.894314 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"e56adeab-d6b1-4e66-8732-8af1bd20ebd5","Type":"ContainerDied","Data":"0ed5b339f769a73202175ced3d09a3215b4ac05314e9f28fa15b9ea6ad3e960a"} Oct 31 00:46:19.894512 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.894397 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"e56adeab-d6b1-4e66-8732-8af1bd20ebd5","Type":"ContainerDied","Data":"f633dbab80c0a149980bba3f0671096e25eacdbbc09c0ab613b947ac7d6e0767"} Oct 31 00:46:19.894512 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.894437 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"e56adeab-d6b1-4e66-8732-8af1bd20ebd5","Type":"ContainerDied","Data":"47515dc25ec84115f53712b6b827624f0fac31e4801faf76867a74573df72162"} Oct 31 00:46:19.894512 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.894464 5274 scope.go:122] "RemoveContainer" containerID="0ed5b339f769a73202175ced3d09a3215b4ac05314e9f28fa15b9ea6ad3e960a" Oct 31 00:46:19.894512 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.894498 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"e56adeab-d6b1-4e66-8732-8af1bd20ebd5","Type":"ContainerDied","Data":"bbf42d9f709388a7850753a73a5f37a9edf59a472dfbeb00303bd08f7b75ca4c"} Oct 31 00:46:19.903324 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.903308 5274 scope.go:122] "RemoveContainer" containerID="f633dbab80c0a149980bba3f0671096e25eacdbbc09c0ab613b947ac7d6e0767" Oct 31 00:46:19.911383 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.911364 5274 scope.go:122] "RemoveContainer" containerID="47515dc25ec84115f53712b6b827624f0fac31e4801faf76867a74573df72162" Oct 31 00:46:19.919473 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.919453 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:46:19.920156 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.920144 5274 scope.go:122] "RemoveContainer" containerID="216d1ab841101a1cf1d7a33d1876ae0da0b95969b8b068c69a459fafc42177d7" Oct 31 00:46:19.923473 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.923453 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:46:19.928330 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.928313 5274 scope.go:122] "RemoveContainer" containerID="0ed5b339f769a73202175ced3d09a3215b4ac05314e9f28fa15b9ea6ad3e960a" Oct 31 00:46:19.928587 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:46:19.928567 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ed5b339f769a73202175ced3d09a3215b4ac05314e9f28fa15b9ea6ad3e960a\": container with ID starting with 0ed5b339f769a73202175ced3d09a3215b4ac05314e9f28fa15b9ea6ad3e960a not found: ID does not exist" containerID="0ed5b339f769a73202175ced3d09a3215b4ac05314e9f28fa15b9ea6ad3e960a" Oct 31 00:46:19.928633 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.928595 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ed5b339f769a73202175ced3d09a3215b4ac05314e9f28fa15b9ea6ad3e960a"} err="failed to get container status \"0ed5b339f769a73202175ced3d09a3215b4ac05314e9f28fa15b9ea6ad3e960a\": rpc error: code = NotFound desc = could not find container \"0ed5b339f769a73202175ced3d09a3215b4ac05314e9f28fa15b9ea6ad3e960a\": container with ID starting with 0ed5b339f769a73202175ced3d09a3215b4ac05314e9f28fa15b9ea6ad3e960a not found: ID does not exist" Oct 31 00:46:19.928633 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.928612 5274 scope.go:122] "RemoveContainer" containerID="f633dbab80c0a149980bba3f0671096e25eacdbbc09c0ab613b947ac7d6e0767" Oct 31 00:46:19.928837 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:46:19.928823 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f633dbab80c0a149980bba3f0671096e25eacdbbc09c0ab613b947ac7d6e0767\": container with ID starting with f633dbab80c0a149980bba3f0671096e25eacdbbc09c0ab613b947ac7d6e0767 not found: ID does not exist" containerID="f633dbab80c0a149980bba3f0671096e25eacdbbc09c0ab613b947ac7d6e0767" Oct 31 00:46:19.928877 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.928840 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f633dbab80c0a149980bba3f0671096e25eacdbbc09c0ab613b947ac7d6e0767"} err="failed to get container status \"f633dbab80c0a149980bba3f0671096e25eacdbbc09c0ab613b947ac7d6e0767\": rpc error: code = NotFound desc = could not find container \"f633dbab80c0a149980bba3f0671096e25eacdbbc09c0ab613b947ac7d6e0767\": container with ID starting with f633dbab80c0a149980bba3f0671096e25eacdbbc09c0ab613b947ac7d6e0767 not found: ID does not exist" Oct 31 00:46:19.928877 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.928851 5274 scope.go:122] "RemoveContainer" containerID="47515dc25ec84115f53712b6b827624f0fac31e4801faf76867a74573df72162" Oct 31 00:46:19.929089 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:46:19.929073 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47515dc25ec84115f53712b6b827624f0fac31e4801faf76867a74573df72162\": container with ID starting with 47515dc25ec84115f53712b6b827624f0fac31e4801faf76867a74573df72162 not found: ID does not exist" containerID="47515dc25ec84115f53712b6b827624f0fac31e4801faf76867a74573df72162" Oct 31 00:46:19.929135 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.929093 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47515dc25ec84115f53712b6b827624f0fac31e4801faf76867a74573df72162"} err="failed to get container status \"47515dc25ec84115f53712b6b827624f0fac31e4801faf76867a74573df72162\": rpc error: code = NotFound desc = could not find container \"47515dc25ec84115f53712b6b827624f0fac31e4801faf76867a74573df72162\": container with ID starting with 47515dc25ec84115f53712b6b827624f0fac31e4801faf76867a74573df72162 not found: ID does not exist" Oct 31 00:46:19.929135 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.929110 5274 scope.go:122] "RemoveContainer" containerID="216d1ab841101a1cf1d7a33d1876ae0da0b95969b8b068c69a459fafc42177d7" Oct 31 00:46:19.929335 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:46:19.929299 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"216d1ab841101a1cf1d7a33d1876ae0da0b95969b8b068c69a459fafc42177d7\": container with ID starting with 216d1ab841101a1cf1d7a33d1876ae0da0b95969b8b068c69a459fafc42177d7 not found: ID does not exist" containerID="216d1ab841101a1cf1d7a33d1876ae0da0b95969b8b068c69a459fafc42177d7" Oct 31 00:46:19.929335 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.929319 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"216d1ab841101a1cf1d7a33d1876ae0da0b95969b8b068c69a459fafc42177d7"} err="failed to get container status \"216d1ab841101a1cf1d7a33d1876ae0da0b95969b8b068c69a459fafc42177d7\": rpc error: code = NotFound desc = could not find container \"216d1ab841101a1cf1d7a33d1876ae0da0b95969b8b068c69a459fafc42177d7\": container with ID starting with 216d1ab841101a1cf1d7a33d1876ae0da0b95969b8b068c69a459fafc42177d7 not found: ID does not exist" Oct 31 00:46:19.929335 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.929331 5274 scope.go:122] "RemoveContainer" containerID="0ed5b339f769a73202175ced3d09a3215b4ac05314e9f28fa15b9ea6ad3e960a" Oct 31 00:46:19.929546 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.929528 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ed5b339f769a73202175ced3d09a3215b4ac05314e9f28fa15b9ea6ad3e960a"} err="failed to get container status \"0ed5b339f769a73202175ced3d09a3215b4ac05314e9f28fa15b9ea6ad3e960a\": rpc error: code = NotFound desc = could not find container \"0ed5b339f769a73202175ced3d09a3215b4ac05314e9f28fa15b9ea6ad3e960a\": container with ID starting with 0ed5b339f769a73202175ced3d09a3215b4ac05314e9f28fa15b9ea6ad3e960a not found: ID does not exist" Oct 31 00:46:19.929592 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.929546 5274 scope.go:122] "RemoveContainer" containerID="f633dbab80c0a149980bba3f0671096e25eacdbbc09c0ab613b947ac7d6e0767" Oct 31 00:46:19.929726 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.929711 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f633dbab80c0a149980bba3f0671096e25eacdbbc09c0ab613b947ac7d6e0767"} err="failed to get container status \"f633dbab80c0a149980bba3f0671096e25eacdbbc09c0ab613b947ac7d6e0767\": rpc error: code = NotFound desc = could not find container \"f633dbab80c0a149980bba3f0671096e25eacdbbc09c0ab613b947ac7d6e0767\": container with ID starting with f633dbab80c0a149980bba3f0671096e25eacdbbc09c0ab613b947ac7d6e0767 not found: ID does not exist" Oct 31 00:46:19.929767 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.929726 5274 scope.go:122] "RemoveContainer" containerID="47515dc25ec84115f53712b6b827624f0fac31e4801faf76867a74573df72162" Oct 31 00:46:19.929910 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.929895 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47515dc25ec84115f53712b6b827624f0fac31e4801faf76867a74573df72162"} err="failed to get container status \"47515dc25ec84115f53712b6b827624f0fac31e4801faf76867a74573df72162\": rpc error: code = NotFound desc = could not find container \"47515dc25ec84115f53712b6b827624f0fac31e4801faf76867a74573df72162\": container with ID starting with 47515dc25ec84115f53712b6b827624f0fac31e4801faf76867a74573df72162 not found: ID does not exist" Oct 31 00:46:19.930055 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.929910 5274 scope.go:122] "RemoveContainer" containerID="216d1ab841101a1cf1d7a33d1876ae0da0b95969b8b068c69a459fafc42177d7" Oct 31 00:46:19.930167 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.930147 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"216d1ab841101a1cf1d7a33d1876ae0da0b95969b8b068c69a459fafc42177d7"} err="failed to get container status \"216d1ab841101a1cf1d7a33d1876ae0da0b95969b8b068c69a459fafc42177d7\": rpc error: code = NotFound desc = could not find container \"216d1ab841101a1cf1d7a33d1876ae0da0b95969b8b068c69a459fafc42177d7\": container with ID starting with 216d1ab841101a1cf1d7a33d1876ae0da0b95969b8b068c69a459fafc42177d7 not found: ID does not exist" Oct 31 00:46:19.930209 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.930168 5274 scope.go:122] "RemoveContainer" containerID="0ed5b339f769a73202175ced3d09a3215b4ac05314e9f28fa15b9ea6ad3e960a" Oct 31 00:46:19.930390 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.930371 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ed5b339f769a73202175ced3d09a3215b4ac05314e9f28fa15b9ea6ad3e960a"} err="failed to get container status \"0ed5b339f769a73202175ced3d09a3215b4ac05314e9f28fa15b9ea6ad3e960a\": rpc error: code = NotFound desc = could not find container \"0ed5b339f769a73202175ced3d09a3215b4ac05314e9f28fa15b9ea6ad3e960a\": container with ID starting with 0ed5b339f769a73202175ced3d09a3215b4ac05314e9f28fa15b9ea6ad3e960a not found: ID does not exist" Oct 31 00:46:19.930462 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.930392 5274 scope.go:122] "RemoveContainer" containerID="f633dbab80c0a149980bba3f0671096e25eacdbbc09c0ab613b947ac7d6e0767" Oct 31 00:46:19.930628 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.930608 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f633dbab80c0a149980bba3f0671096e25eacdbbc09c0ab613b947ac7d6e0767"} err="failed to get container status \"f633dbab80c0a149980bba3f0671096e25eacdbbc09c0ab613b947ac7d6e0767\": rpc error: code = NotFound desc = could not find container \"f633dbab80c0a149980bba3f0671096e25eacdbbc09c0ab613b947ac7d6e0767\": container with ID starting with f633dbab80c0a149980bba3f0671096e25eacdbbc09c0ab613b947ac7d6e0767 not found: ID does not exist" Oct 31 00:46:19.930674 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.930629 5274 scope.go:122] "RemoveContainer" containerID="47515dc25ec84115f53712b6b827624f0fac31e4801faf76867a74573df72162" Oct 31 00:46:19.930806 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.930789 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47515dc25ec84115f53712b6b827624f0fac31e4801faf76867a74573df72162"} err="failed to get container status \"47515dc25ec84115f53712b6b827624f0fac31e4801faf76867a74573df72162\": rpc error: code = NotFound desc = could not find container \"47515dc25ec84115f53712b6b827624f0fac31e4801faf76867a74573df72162\": container with ID starting with 47515dc25ec84115f53712b6b827624f0fac31e4801faf76867a74573df72162 not found: ID does not exist" Oct 31 00:46:19.930845 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.930807 5274 scope.go:122] "RemoveContainer" containerID="216d1ab841101a1cf1d7a33d1876ae0da0b95969b8b068c69a459fafc42177d7" Oct 31 00:46:19.931018 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:19.930995 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"216d1ab841101a1cf1d7a33d1876ae0da0b95969b8b068c69a459fafc42177d7"} err="failed to get container status \"216d1ab841101a1cf1d7a33d1876ae0da0b95969b8b068c69a459fafc42177d7\": rpc error: code = NotFound desc = could not find container \"216d1ab841101a1cf1d7a33d1876ae0da0b95969b8b068c69a459fafc42177d7\": container with ID starting with 216d1ab841101a1cf1d7a33d1876ae0da0b95969b8b068c69a459fafc42177d7 not found: ID does not exist" Oct 31 00:46:20.658434 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:20.658403 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:46:20.658675 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:20.658647 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="07f5e099-eae5-418f-864c-1b1fd079b595" containerName="otc-container" containerID="cri-o://5d8beb64989135944f02e20ba5a3bec199cc89ff79a03d3bf13ca5dc804b46c3" gracePeriod=30 Oct 31 00:46:20.899293 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:46:20.899259 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:46:21.481498 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:21.481468 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e56adeab-d6b1-4e66-8732-8af1bd20ebd5" path="/var/lib/kubelet/pods/e56adeab-d6b1-4e66-8732-8af1bd20ebd5/volumes" Oct 31 00:46:22.911361 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:22.911332 5274 generic.go:356] "Generic (PLEG): container finished" podID="07f5e099-eae5-418f-864c-1b1fd079b595" containerID="5d8beb64989135944f02e20ba5a3bec199cc89ff79a03d3bf13ca5dc804b46c3" exitCode=0 Oct 31 00:46:22.911804 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:22.911388 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"07f5e099-eae5-418f-864c-1b1fd079b595","Type":"ContainerDied","Data":"5d8beb64989135944f02e20ba5a3bec199cc89ff79a03d3bf13ca5dc804b46c3"} Oct 31 00:46:23.001378 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.001350 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:23.005643 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.005624 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07f5e099-eae5-418f-864c-1b1fd079b595-kube-api-access-szlb5\" (UniqueName: \"kubernetes.io/projected/07f5e099-eae5-418f-864c-1b1fd079b595-kube-api-access-szlb5\") pod \"07f5e099-eae5-418f-864c-1b1fd079b595\" (UID: \"07f5e099-eae5-418f-864c-1b1fd079b595\") " Oct 31 00:46:23.005735 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.005666 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07f5e099-eae5-418f-864c-1b1fd079b595-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/07f5e099-eae5-418f-864c-1b1fd079b595-data-science-collector-ta-client-cert\") pod \"07f5e099-eae5-418f-864c-1b1fd079b595\" (UID: \"07f5e099-eae5-418f-864c-1b1fd079b595\") " Oct 31 00:46:23.005735 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.005690 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07f5e099-eae5-418f-864c-1b1fd079b595-otc-internal\" (UniqueName: \"kubernetes.io/configmap/07f5e099-eae5-418f-864c-1b1fd079b595-otc-internal\") pod \"07f5e099-eae5-418f-864c-1b1fd079b595\" (UID: \"07f5e099-eae5-418f-864c-1b1fd079b595\") " Oct 31 00:46:23.005735 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.005709 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07f5e099-eae5-418f-864c-1b1fd079b595-tls-certs\" (UniqueName: \"kubernetes.io/secret/07f5e099-eae5-418f-864c-1b1fd079b595-tls-certs\") pod \"07f5e099-eae5-418f-864c-1b1fd079b595\" (UID: \"07f5e099-eae5-418f-864c-1b1fd079b595\") " Oct 31 00:46:23.005899 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.005742 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07f5e099-eae5-418f-864c-1b1fd079b595-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/07f5e099-eae5-418f-864c-1b1fd079b595-otlp-tls-certs\") pod \"07f5e099-eae5-418f-864c-1b1fd079b595\" (UID: \"07f5e099-eae5-418f-864c-1b1fd079b595\") " Oct 31 00:46:23.006130 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.006105 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07f5e099-eae5-418f-864c-1b1fd079b595-otc-internal" pod "07f5e099-eae5-418f-864c-1b1fd079b595" (UID: "07f5e099-eae5-418f-864c-1b1fd079b595"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:46:23.007913 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.007887 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07f5e099-eae5-418f-864c-1b1fd079b595-otlp-tls-certs" pod "07f5e099-eae5-418f-864c-1b1fd079b595" (UID: "07f5e099-eae5-418f-864c-1b1fd079b595"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:23.007913 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.007900 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07f5e099-eae5-418f-864c-1b1fd079b595-tls-certs" pod "07f5e099-eae5-418f-864c-1b1fd079b595" (UID: "07f5e099-eae5-418f-864c-1b1fd079b595"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:23.008089 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.008049 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07f5e099-eae5-418f-864c-1b1fd079b595-data-science-collector-ta-client-cert" pod "07f5e099-eae5-418f-864c-1b1fd079b595" (UID: "07f5e099-eae5-418f-864c-1b1fd079b595"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:23.008148 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.008131 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07f5e099-eae5-418f-864c-1b1fd079b595-kube-api-access-szlb5" pod "07f5e099-eae5-418f-864c-1b1fd079b595" (UID: "07f5e099-eae5-418f-864c-1b1fd079b595"). InnerVolumeSpecName "kube-api-access-szlb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:46:23.106209 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.106135 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07f5e099-eae5-418f-864c-1b1fd079b595-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/07f5e099-eae5-418f-864c-1b1fd079b595-monitoring-tls-certs\") pod \"07f5e099-eae5-418f-864c-1b1fd079b595\" (UID: \"07f5e099-eae5-418f-864c-1b1fd079b595\") " Oct 31 00:46:23.106358 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.106342 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szlb5\" (UniqueName: \"kubernetes.io/projected/07f5e099-eae5-418f-864c-1b1fd079b595-kube-api-access-szlb5\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:23.106418 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.106365 5274 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/07f5e099-eae5-418f-864c-1b1fd079b595-data-science-collector-ta-client-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:23.106418 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.106379 5274 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/07f5e099-eae5-418f-864c-1b1fd079b595-otc-internal\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:23.106418 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.106393 5274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07f5e099-eae5-418f-864c-1b1fd079b595-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:23.106418 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.106407 5274 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/07f5e099-eae5-418f-864c-1b1fd079b595-otlp-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:23.108087 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.108068 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07f5e099-eae5-418f-864c-1b1fd079b595-monitoring-tls-certs" pod "07f5e099-eae5-418f-864c-1b1fd079b595" (UID: "07f5e099-eae5-418f-864c-1b1fd079b595"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:23.207271 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.207248 5274 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/07f5e099-eae5-418f-864c-1b1fd079b595-monitoring-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:23.917402 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.917373 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:23.917402 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.917387 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"07f5e099-eae5-418f-864c-1b1fd079b595","Type":"ContainerDied","Data":"1c1939a687ce24c9d2d4584b6c3678066df474d035f661381797d84c0a146fa2"} Oct 31 00:46:23.917826 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.917438 5274 scope.go:122] "RemoveContainer" containerID="5d8beb64989135944f02e20ba5a3bec199cc89ff79a03d3bf13ca5dc804b46c3" Oct 31 00:46:23.937212 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.937186 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:46:23.943436 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.943415 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:46:23.970352 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.970333 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:46:23.970656 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.970645 5274 cpu_manager.go:414] "containerMap: removing container" podUID="e56adeab-d6b1-4e66-8732-8af1bd20ebd5" containerName="config-reloader" Oct 31 00:46:23.970705 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.970658 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e56adeab-d6b1-4e66-8732-8af1bd20ebd5" containerName="config-reloader" Oct 31 00:46:23.970705 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.970677 5274 cpu_manager.go:414] "containerMap: removing container" podUID="07f5e099-eae5-418f-864c-1b1fd079b595" containerName="otc-container" Oct 31 00:46:23.970705 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.970682 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07f5e099-eae5-418f-864c-1b1fd079b595" containerName="otc-container" Oct 31 00:46:23.970705 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.970692 5274 cpu_manager.go:414] "containerMap: removing container" podUID="e56adeab-d6b1-4e66-8732-8af1bd20ebd5" containerName="init-config-reloader" Oct 31 00:46:23.970705 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.970698 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e56adeab-d6b1-4e66-8732-8af1bd20ebd5" containerName="init-config-reloader" Oct 31 00:46:23.970860 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.970711 5274 cpu_manager.go:414] "containerMap: removing container" podUID="e56adeab-d6b1-4e66-8732-8af1bd20ebd5" containerName="prometheus" Oct 31 00:46:23.970860 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.970715 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e56adeab-d6b1-4e66-8732-8af1bd20ebd5" containerName="prometheus" Oct 31 00:46:23.970860 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.970726 5274 cpu_manager.go:414] "containerMap: removing container" podUID="e56adeab-d6b1-4e66-8732-8af1bd20ebd5" containerName="thanos-sidecar" Oct 31 00:46:23.970860 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.970730 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e56adeab-d6b1-4e66-8732-8af1bd20ebd5" containerName="thanos-sidecar" Oct 31 00:46:23.970860 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.970788 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="e56adeab-d6b1-4e66-8732-8af1bd20ebd5" containerName="prometheus" Oct 31 00:46:23.970860 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.970797 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="e56adeab-d6b1-4e66-8732-8af1bd20ebd5" containerName="thanos-sidecar" Oct 31 00:46:23.970860 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.970805 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="e56adeab-d6b1-4e66-8732-8af1bd20ebd5" containerName="config-reloader" Oct 31 00:46:23.970860 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.970812 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="07f5e099-eae5-418f-864c-1b1fd079b595" containerName="otc-container" Oct 31 00:46:23.975114 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.975091 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:23.979594 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.979572 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-7hrqp\"" Oct 31 00:46:23.979690 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.979661 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:46:23.979740 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.979727 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:46:23.980029 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.980008 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:46:23.980082 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.980016 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:46:23.982457 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.982436 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:46:23.987596 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:23.987576 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:46:24.012847 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:24.012818 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"ca87bc1b-de22-4fc6-96db-38cafc8de9fa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:24.013031 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:24.012863 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhd9r\" (UniqueName: \"kubernetes.io/projected/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-kube-api-access-rhd9r\") pod \"data-science-collector-collector-0\" (UID: \"ca87bc1b-de22-4fc6-96db-38cafc8de9fa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:24.013031 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:24.012880 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ca87bc1b-de22-4fc6-96db-38cafc8de9fa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:24.013160 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:24.013054 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"ca87bc1b-de22-4fc6-96db-38cafc8de9fa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:24.013160 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:24.013084 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ca87bc1b-de22-4fc6-96db-38cafc8de9fa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:24.013160 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:24.013122 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ca87bc1b-de22-4fc6-96db-38cafc8de9fa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:24.114163 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:24.114088 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhd9r\" (UniqueName: \"kubernetes.io/projected/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-kube-api-access-rhd9r\") pod \"data-science-collector-collector-0\" (UID: \"ca87bc1b-de22-4fc6-96db-38cafc8de9fa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:24.114163 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:24.114132 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ca87bc1b-de22-4fc6-96db-38cafc8de9fa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:24.114352 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:24.114180 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"ca87bc1b-de22-4fc6-96db-38cafc8de9fa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:24.114352 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:24.114203 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ca87bc1b-de22-4fc6-96db-38cafc8de9fa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:24.114352 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:24.114335 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ca87bc1b-de22-4fc6-96db-38cafc8de9fa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:24.114503 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:24.114396 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"ca87bc1b-de22-4fc6-96db-38cafc8de9fa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:24.115182 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:24.115156 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"ca87bc1b-de22-4fc6-96db-38cafc8de9fa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:24.116799 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:24.116767 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ca87bc1b-de22-4fc6-96db-38cafc8de9fa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:24.116903 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:24.116804 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ca87bc1b-de22-4fc6-96db-38cafc8de9fa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:24.116903 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:24.116812 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"ca87bc1b-de22-4fc6-96db-38cafc8de9fa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:24.116903 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:24.116889 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ca87bc1b-de22-4fc6-96db-38cafc8de9fa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:24.126820 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:24.126797 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhd9r\" (UniqueName: \"kubernetes.io/projected/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-kube-api-access-rhd9r\") pod \"data-science-collector-collector-0\" (UID: \"ca87bc1b-de22-4fc6-96db-38cafc8de9fa\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:24.288596 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:24.288566 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:24.567341 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:24.567317 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:46:24.569415 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:46:24.569390 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca87bc1b_de22_4fc6_96db_38cafc8de9fa.slice/crio-1fa5c5a0cf78074e46884d8f83ad45ecd0622eb1fc8327e2428e1cb1d056cb28 WatchSource:0}: Error finding container 1fa5c5a0cf78074e46884d8f83ad45ecd0622eb1fc8327e2428e1cb1d056cb28: Status 404 returned error can't find the container with id 1fa5c5a0cf78074e46884d8f83ad45ecd0622eb1fc8327e2428e1cb1d056cb28 Oct 31 00:46:24.571748 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:24.571712 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:46:24.571843 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:24.571785 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:46:24.924368 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:24.924338 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ca87bc1b-de22-4fc6-96db-38cafc8de9fa","Type":"ContainerStarted","Data":"1e0413e85ca01edb214e1849303a1dd816c9532d79a871a61f4629d1f30a3778"} Oct 31 00:46:24.924860 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:24.924399 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ca87bc1b-de22-4fc6-96db-38cafc8de9fa","Type":"ContainerStarted","Data":"1fa5c5a0cf78074e46884d8f83ad45ecd0622eb1fc8327e2428e1cb1d056cb28"} Oct 31 00:46:24.947590 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:24.947506 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.947492443 podStartE2EDuration="1.947492443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:24.94624678 +0000 UTC m=+2821.583065495" watchObservedRunningTime="2026-10-31 00:46:24.947492443 +0000 UTC m=+2821.584311159" Oct 31 00:46:25.481464 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:25.481428 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07f5e099-eae5-418f-864c-1b1fd079b595" path="/var/lib/kubelet/pods/07f5e099-eae5-418f-864c-1b1fd079b595/volumes" Oct 31 00:46:30.958373 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:30.958342 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:46:30.962846 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:30.962816 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:30.967092 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:30.967070 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 31 00:46:30.967092 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:30.967071 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:46:30.967994 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:30.967928 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 31 00:46:30.967994 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:30.967985 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 31 00:46:30.968194 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:30.968003 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:46:30.968194 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:30.968010 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 31 00:46:30.968194 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:30.968012 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 31 00:46:30.968194 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:30.968004 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-b6hd7\"" Oct 31 00:46:30.968194 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:30.968064 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 31 00:46:30.978756 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:30.978733 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:46:31.067263 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.067234 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/434094cd-527e-4be5-bd9d-0a49be98a7bf-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:31.067263 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.067262 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/434094cd-527e-4be5-bd9d-0a49be98a7bf-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:31.067499 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.067283 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/434094cd-527e-4be5-bd9d-0a49be98a7bf-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:31.067499 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.067307 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/434094cd-527e-4be5-bd9d-0a49be98a7bf-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:31.067499 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.067365 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:31.067499 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.067464 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:31.067653 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.067504 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/434094cd-527e-4be5-bd9d-0a49be98a7bf-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:31.067653 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.067569 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:31.067653 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.067609 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:31.067653 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.067631 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:31.067816 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.067656 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6tnj\" (UniqueName: \"kubernetes.io/projected/434094cd-527e-4be5-bd9d-0a49be98a7bf-kube-api-access-x6tnj\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:31.067816 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.067725 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:31.168814 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.168784 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6tnj\" (UniqueName: \"kubernetes.io/projected/434094cd-527e-4be5-bd9d-0a49be98a7bf-kube-api-access-x6tnj\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:31.168929 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.168823 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:31.168929 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.168847 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/434094cd-527e-4be5-bd9d-0a49be98a7bf-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:31.168929 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.168863 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/434094cd-527e-4be5-bd9d-0a49be98a7bf-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:31.168929 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.168892 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/434094cd-527e-4be5-bd9d-0a49be98a7bf-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:31.168929 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.168917 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/434094cd-527e-4be5-bd9d-0a49be98a7bf-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:31.169223 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.168934 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:31.169223 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.168997 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:31.169223 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.169028 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/434094cd-527e-4be5-bd9d-0a49be98a7bf-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:31.169606 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.169578 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/434094cd-527e-4be5-bd9d-0a49be98a7bf-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:31.169705 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.169651 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:31.169763 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.169702 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:31.169763 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.169705 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/434094cd-527e-4be5-bd9d-0a49be98a7bf-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:31.169763 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.169749 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:31.169922 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.169830 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/434094cd-527e-4be5-bd9d-0a49be98a7bf-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:31.172142 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.172118 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/434094cd-527e-4be5-bd9d-0a49be98a7bf-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:31.172245 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.172193 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:31.172364 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.172341 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/434094cd-527e-4be5-bd9d-0a49be98a7bf-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:31.172465 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.172426 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:31.172533 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.172514 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:31.172711 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.172687 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:31.172817 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.172802 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:31.173056 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.173040 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:31.181682 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.181659 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6tnj\" (UniqueName: \"kubernetes.io/projected/434094cd-527e-4be5-bd9d-0a49be98a7bf-kube-api-access-x6tnj\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:31.713727 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.713684 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:46:31.713932 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.713907 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="ca87bc1b-de22-4fc6-96db-38cafc8de9fa" containerName="otc-container" containerID="cri-o://1e0413e85ca01edb214e1849303a1dd816c9532d79a871a61f4629d1f30a3778" gracePeriod=30 Oct 31 00:46:31.959873 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.959846 5274 generic.go:356] "Generic (PLEG): container finished" podID="ca87bc1b-de22-4fc6-96db-38cafc8de9fa" containerID="1e0413e85ca01edb214e1849303a1dd816c9532d79a871a61f4629d1f30a3778" exitCode=0 Oct 31 00:46:31.960205 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.959914 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ca87bc1b-de22-4fc6-96db-38cafc8de9fa","Type":"ContainerDied","Data":"1e0413e85ca01edb214e1849303a1dd816c9532d79a871a61f4629d1f30a3778"} Oct 31 00:46:31.960205 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.959984 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ca87bc1b-de22-4fc6-96db-38cafc8de9fa","Type":"ContainerDied","Data":"1fa5c5a0cf78074e46884d8f83ad45ecd0622eb1fc8327e2428e1cb1d056cb28"} Oct 31 00:46:31.960205 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.960005 5274 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fa5c5a0cf78074e46884d8f83ad45ecd0622eb1fc8327e2428e1cb1d056cb28" Oct 31 00:46:31.961232 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.961217 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:31.977355 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.977300 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-monitoring-tls-certs\") pod \"ca87bc1b-de22-4fc6-96db-38cafc8de9fa\" (UID: \"ca87bc1b-de22-4fc6-96db-38cafc8de9fa\") " Oct 31 00:46:31.977355 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.977346 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-otlp-tls-certs\") pod \"ca87bc1b-de22-4fc6-96db-38cafc8de9fa\" (UID: \"ca87bc1b-de22-4fc6-96db-38cafc8de9fa\") " Oct 31 00:46:31.977502 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.977412 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-tls-certs\") pod \"ca87bc1b-de22-4fc6-96db-38cafc8de9fa\" (UID: \"ca87bc1b-de22-4fc6-96db-38cafc8de9fa\") " Oct 31 00:46:31.977502 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.977447 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-data-science-collector-ta-client-cert\") pod \"ca87bc1b-de22-4fc6-96db-38cafc8de9fa\" (UID: \"ca87bc1b-de22-4fc6-96db-38cafc8de9fa\") " Oct 31 00:46:31.977502 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.977490 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-otc-internal\" (UniqueName: \"kubernetes.io/configmap/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-otc-internal\") pod \"ca87bc1b-de22-4fc6-96db-38cafc8de9fa\" (UID: \"ca87bc1b-de22-4fc6-96db-38cafc8de9fa\") " Oct 31 00:46:31.977655 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.977521 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-kube-api-access-rhd9r\" (UniqueName: \"kubernetes.io/projected/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-kube-api-access-rhd9r\") pod \"ca87bc1b-de22-4fc6-96db-38cafc8de9fa\" (UID: \"ca87bc1b-de22-4fc6-96db-38cafc8de9fa\") " Oct 31 00:46:31.977910 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.977871 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-otc-internal" pod "ca87bc1b-de22-4fc6-96db-38cafc8de9fa" (UID: "ca87bc1b-de22-4fc6-96db-38cafc8de9fa"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:46:31.979620 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.979595 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-otlp-tls-certs" pod "ca87bc1b-de22-4fc6-96db-38cafc8de9fa" (UID: "ca87bc1b-de22-4fc6-96db-38cafc8de9fa"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:31.979620 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.979614 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-tls-certs" pod "ca87bc1b-de22-4fc6-96db-38cafc8de9fa" (UID: "ca87bc1b-de22-4fc6-96db-38cafc8de9fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:31.979877 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.979859 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-data-science-collector-ta-client-cert" pod "ca87bc1b-de22-4fc6-96db-38cafc8de9fa" (UID: "ca87bc1b-de22-4fc6-96db-38cafc8de9fa"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:31.979940 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.979887 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-monitoring-tls-certs" pod "ca87bc1b-de22-4fc6-96db-38cafc8de9fa" (UID: "ca87bc1b-de22-4fc6-96db-38cafc8de9fa"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:31.980016 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:31.979976 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-kube-api-access-rhd9r" pod "ca87bc1b-de22-4fc6-96db-38cafc8de9fa" (UID: "ca87bc1b-de22-4fc6-96db-38cafc8de9fa"). InnerVolumeSpecName "kube-api-access-rhd9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:46:32.078898 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:32.078868 5274 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-monitoring-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:32.078898 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:32.078893 5274 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-otlp-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:32.078898 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:32.078903 5274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:32.079143 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:32.078914 5274 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-data-science-collector-ta-client-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:32.079143 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:32.078923 5274 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-otc-internal\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:32.079143 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:32.078932 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhd9r\" (UniqueName: \"kubernetes.io/projected/ca87bc1b-de22-4fc6-96db-38cafc8de9fa-kube-api-access-rhd9r\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:32.965162 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:32.965136 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:32.988257 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:32.988223 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:46:32.993147 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:32.993123 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:46:33.020876 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:33.020850 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:46:33.021260 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:33.021247 5274 cpu_manager.go:414] "containerMap: removing container" podUID="ca87bc1b-de22-4fc6-96db-38cafc8de9fa" containerName="otc-container" Oct 31 00:46:33.021311 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:33.021262 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca87bc1b-de22-4fc6-96db-38cafc8de9fa" containerName="otc-container" Oct 31 00:46:33.021357 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:33.021333 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca87bc1b-de22-4fc6-96db-38cafc8de9fa" containerName="otc-container" Oct 31 00:46:33.024809 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:33.024791 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:33.029103 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:33.029073 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-7hrqp\"" Oct 31 00:46:33.029103 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:33.029099 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:46:33.029254 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:33.029135 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:46:33.029254 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:33.029197 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:46:33.029508 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:33.029494 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:46:33.029664 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:33.029650 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:46:33.035505 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:33.035486 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:46:33.087895 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:33.087868 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/67ff5c67-5207-49be-93f2-81429cd1d8a9-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"67ff5c67-5207-49be-93f2-81429cd1d8a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:33.088041 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:33.087899 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/67ff5c67-5207-49be-93f2-81429cd1d8a9-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"67ff5c67-5207-49be-93f2-81429cd1d8a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:33.088041 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:33.087926 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/67ff5c67-5207-49be-93f2-81429cd1d8a9-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"67ff5c67-5207-49be-93f2-81429cd1d8a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:33.088041 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:33.087948 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67ff5c67-5207-49be-93f2-81429cd1d8a9-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"67ff5c67-5207-49be-93f2-81429cd1d8a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:33.088150 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:33.088082 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/67ff5c67-5207-49be-93f2-81429cd1d8a9-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"67ff5c67-5207-49be-93f2-81429cd1d8a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:33.088150 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:33.088118 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hdgz\" (UniqueName: \"kubernetes.io/projected/67ff5c67-5207-49be-93f2-81429cd1d8a9-kube-api-access-8hdgz\") pod \"data-science-collector-collector-0\" (UID: \"67ff5c67-5207-49be-93f2-81429cd1d8a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:33.189087 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:33.189057 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hdgz\" (UniqueName: \"kubernetes.io/projected/67ff5c67-5207-49be-93f2-81429cd1d8a9-kube-api-access-8hdgz\") pod \"data-science-collector-collector-0\" (UID: \"67ff5c67-5207-49be-93f2-81429cd1d8a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:33.189219 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:33.189141 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/67ff5c67-5207-49be-93f2-81429cd1d8a9-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"67ff5c67-5207-49be-93f2-81429cd1d8a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:33.189219 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:33.189165 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/67ff5c67-5207-49be-93f2-81429cd1d8a9-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"67ff5c67-5207-49be-93f2-81429cd1d8a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:33.189219 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:33.189190 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/67ff5c67-5207-49be-93f2-81429cd1d8a9-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"67ff5c67-5207-49be-93f2-81429cd1d8a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:33.189219 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:33.189211 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67ff5c67-5207-49be-93f2-81429cd1d8a9-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"67ff5c67-5207-49be-93f2-81429cd1d8a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:33.189446 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:33.189286 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/67ff5c67-5207-49be-93f2-81429cd1d8a9-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"67ff5c67-5207-49be-93f2-81429cd1d8a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:33.189877 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:33.189857 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/67ff5c67-5207-49be-93f2-81429cd1d8a9-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"67ff5c67-5207-49be-93f2-81429cd1d8a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:33.191587 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:33.191559 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/67ff5c67-5207-49be-93f2-81429cd1d8a9-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"67ff5c67-5207-49be-93f2-81429cd1d8a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:33.191696 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:33.191676 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/67ff5c67-5207-49be-93f2-81429cd1d8a9-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"67ff5c67-5207-49be-93f2-81429cd1d8a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:33.192134 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:33.192109 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67ff5c67-5207-49be-93f2-81429cd1d8a9-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"67ff5c67-5207-49be-93f2-81429cd1d8a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:33.192180 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:33.192136 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/67ff5c67-5207-49be-93f2-81429cd1d8a9-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"67ff5c67-5207-49be-93f2-81429cd1d8a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:33.202331 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:33.202309 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hdgz\" (UniqueName: \"kubernetes.io/projected/67ff5c67-5207-49be-93f2-81429cd1d8a9-kube-api-access-8hdgz\") pod \"data-science-collector-collector-0\" (UID: \"67ff5c67-5207-49be-93f2-81429cd1d8a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:33.339356 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:33.339333 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:46:33.482199 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:33.482136 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca87bc1b-de22-4fc6-96db-38cafc8de9fa" path="/var/lib/kubelet/pods/ca87bc1b-de22-4fc6-96db-38cafc8de9fa/volumes" Oct 31 00:46:33.613856 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:33.613835 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:46:33.616043 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:46:33.616016 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ff5c67_5207_49be_93f2_81429cd1d8a9.slice/crio-cc4a318ee664d7356a0a958033121a444484aa2a3dd20f358e2a21d5f6acc79b WatchSource:0}: Error finding container cc4a318ee664d7356a0a958033121a444484aa2a3dd20f358e2a21d5f6acc79b: Status 404 returned error can't find the container with id cc4a318ee664d7356a0a958033121a444484aa2a3dd20f358e2a21d5f6acc79b Oct 31 00:46:33.618060 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:33.618009 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:46:33.618139 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:33.618086 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:46:33.971487 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:33.971451 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"67ff5c67-5207-49be-93f2-81429cd1d8a9","Type":"ContainerStarted","Data":"1a756d003be069795fbca067b851999389e0a68a198e7e0dcd7521ffde128941"} Oct 31 00:46:33.971851 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:33.971500 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"67ff5c67-5207-49be-93f2-81429cd1d8a9","Type":"ContainerStarted","Data":"cc4a318ee664d7356a0a958033121a444484aa2a3dd20f358e2a21d5f6acc79b"} Oct 31 00:46:33.994152 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:33.994105 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=0.994092158 podStartE2EDuration="994.092158ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:33.992779853 +0000 UTC m=+2830.629598573" watchObservedRunningTime="2026-10-31 00:46:33.994092158 +0000 UTC m=+2830.630910873" Oct 31 00:46:40.650020 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:40.649987 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:40.657022 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:40.656995 5274 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:40.750928 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:40.750901 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:40.751078 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:40.751023 5274 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:40.754558 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:40.754541 5274 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") DevicePath \"csi-3beb91b3ebf177571ca474589da3560cd18787cfa50b2a0b9bfef85661e6cf21\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:40.845405 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:40.845377 5274 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/534eb232558c8505fa55558aa87cc96e2d2b0adb946edb717939ec387be54e97/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:40.860819 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:40.860798 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:40.883659 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:40.883636 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:41.168854 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:41.168827 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:46:41.170388 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:46:41.170362 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod434094cd_527e_4be5_bd9d_0a49be98a7bf.slice/crio-1a9459be232d74b0cfb54f87d01f8987b01697af28cf05cb70d7594b92a7d2bc WatchSource:0}: Error finding container 1a9459be232d74b0cfb54f87d01f8987b01697af28cf05cb70d7594b92a7d2bc: Status 404 returned error can't find the container with id 1a9459be232d74b0cfb54f87d01f8987b01697af28cf05cb70d7594b92a7d2bc Oct 31 00:46:42.010645 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:42.010614 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"434094cd-527e-4be5-bd9d-0a49be98a7bf","Type":"ContainerStarted","Data":"8e33114c22bf125e2825dd32ed66068d72bc92e69e7e5c09b5bac4066622219d"} Oct 31 00:46:42.011032 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:42.010675 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"434094cd-527e-4be5-bd9d-0a49be98a7bf","Type":"ContainerStarted","Data":"1a9459be232d74b0cfb54f87d01f8987b01697af28cf05cb70d7594b92a7d2bc"} Oct 31 00:46:46.033013 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:46.032976 5274 generic.go:356] "Generic (PLEG): container finished" podID="434094cd-527e-4be5-bd9d-0a49be98a7bf" containerID="8e33114c22bf125e2825dd32ed66068d72bc92e69e7e5c09b5bac4066622219d" exitCode=0 Oct 31 00:46:46.033371 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:46.033025 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"434094cd-527e-4be5-bd9d-0a49be98a7bf","Type":"ContainerDied","Data":"8e33114c22bf125e2825dd32ed66068d72bc92e69e7e5c09b5bac4066622219d"} Oct 31 00:46:47.041111 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:47.041079 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"434094cd-527e-4be5-bd9d-0a49be98a7bf","Type":"ContainerStarted","Data":"609efa17c1c4aa2ea92704a81291fef031aa600182cd27375f5c47da88de1336"} Oct 31 00:46:47.041466 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:47.041145 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"434094cd-527e-4be5-bd9d-0a49be98a7bf","Type":"ContainerStarted","Data":"a6375fe434542308f81abc9825ac4af4f42e514f825cabcd801ec18fe63cbaba"} Oct 31 00:46:47.041466 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:47.041173 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"434094cd-527e-4be5-bd9d-0a49be98a7bf","Type":"ContainerStarted","Data":"16ae09a3d99b6c737b11927ca8ebfa6e81ef6c2467de0f43c4adb57bd3782694"} Oct 31 00:46:47.075268 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:47.075219 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=13.589512285 podStartE2EDuration="17.075205305s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.48569302s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:47.07284978 +0000 UTC m=+2843.709668498" watchObservedRunningTime="2026-10-31 00:46:47.075205305 +0000 UTC m=+2843.712024030" Oct 31 00:46:50.884280 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:50.884246 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:50.898803 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:46:50.898774 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:46:55.883830 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:55.883796 5274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:55.888869 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:55.888851 5274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:46:56.089741 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:46:56.089713 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:07.870431 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:07.870392 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:47:07.870859 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:07.870640 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="67ff5c67-5207-49be-93f2-81429cd1d8a9" containerName="otc-container" containerID="cri-o://1a756d003be069795fbca067b851999389e0a68a198e7e0dcd7521ffde128941" gracePeriod=30 Oct 31 00:47:09.288116 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:09.288076 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-n684d"] Oct 31 00:47:09.288601 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:09.288300 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-n684d" podUID="cdbf3648-1abd-4d7b-959d-637a1e564adf" containerName="korrel8r" containerID="cri-o://54f456929e1f583631a2e24f97d945cbc62ca6471233d20880b808022bc19e30" gracePeriod=30 Oct 31 00:47:09.290226 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:09.290200 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-mpkxh"] Oct 31 00:47:09.290380 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:09.290361 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-mpkxh" podUID="644488ea-9480-4d33-9e58-701b46b95a13" containerName="kube-rbac-proxy" containerID="cri-o://23a18d03209ee6ee91728c4e12eb7e7224323ee51b16c3955a3ef61fad2a27a9" gracePeriod=30 Oct 31 00:47:09.507528 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:09.507497 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:47:09.508205 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:09.507796 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="3f69d683-62af-468f-8ae5-f2b2d101bf61" containerName="alertmanager" containerID="cri-o://2273a440ff8b094cfffc936bf77205754ee60a3b4410bfc11ee6d365776f13ab" gracePeriod=120 Oct 31 00:47:09.508205 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:09.507823 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="3f69d683-62af-468f-8ae5-f2b2d101bf61" containerName="config-reloader" containerID="cri-o://eb445f800eb5fda3011cc89baf7988dce7a671257953e71fa9453edb67b6b314" gracePeriod=120 Oct 31 00:47:09.518243 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:09.518216 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:47:09.518540 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:09.518515 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="434094cd-527e-4be5-bd9d-0a49be98a7bf" containerName="prometheus" containerID="cri-o://16ae09a3d99b6c737b11927ca8ebfa6e81ef6c2467de0f43c4adb57bd3782694" gracePeriod=600 Oct 31 00:47:09.518633 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:09.518541 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="434094cd-527e-4be5-bd9d-0a49be98a7bf" containerName="thanos-sidecar" containerID="cri-o://609efa17c1c4aa2ea92704a81291fef031aa600182cd27375f5c47da88de1336" gracePeriod=600 Oct 31 00:47:09.518697 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:09.518640 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="434094cd-527e-4be5-bd9d-0a49be98a7bf" containerName="config-reloader" containerID="cri-o://a6375fe434542308f81abc9825ac4af4f42e514f825cabcd801ec18fe63cbaba" gracePeriod=600 Oct 31 00:47:09.643456 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:09.643374 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-mpkxh" Oct 31 00:47:09.645862 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:09.645836 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-n684d" Oct 31 00:47:09.679527 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:09.679492 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/644488ea-9480-4d33-9e58-701b46b95a13-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/644488ea-9480-4d33-9e58-701b46b95a13-prometheus-client-cert\") pod \"644488ea-9480-4d33-9e58-701b46b95a13\" (UID: \"644488ea-9480-4d33-9e58-701b46b95a13\") " Oct 31 00:47:09.680764 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:09.679537 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/644488ea-9480-4d33-9e58-701b46b95a13-kube-api-access-66djb\" (UniqueName: \"kubernetes.io/projected/644488ea-9480-4d33-9e58-701b46b95a13-kube-api-access-66djb\") pod \"644488ea-9480-4d33-9e58-701b46b95a13\" (UID: \"644488ea-9480-4d33-9e58-701b46b95a13\") " Oct 31 00:47:09.680764 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:09.679584 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdbf3648-1abd-4d7b-959d-637a1e564adf-config\" (UniqueName: \"kubernetes.io/configmap/cdbf3648-1abd-4d7b-959d-637a1e564adf-config\") pod \"cdbf3648-1abd-4d7b-959d-637a1e564adf\" (UID: \"cdbf3648-1abd-4d7b-959d-637a1e564adf\") " Oct 31 00:47:09.680764 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:09.679611 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/644488ea-9480-4d33-9e58-701b46b95a13-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/644488ea-9480-4d33-9e58-701b46b95a13-kube-rbac-proxy-config\") pod \"644488ea-9480-4d33-9e58-701b46b95a13\" (UID: \"644488ea-9480-4d33-9e58-701b46b95a13\") " Oct 31 00:47:09.680764 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:09.679660 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/644488ea-9480-4d33-9e58-701b46b95a13-tls\" (UniqueName: \"kubernetes.io/secret/644488ea-9480-4d33-9e58-701b46b95a13-tls\") pod \"644488ea-9480-4d33-9e58-701b46b95a13\" (UID: \"644488ea-9480-4d33-9e58-701b46b95a13\") " Oct 31 00:47:09.680764 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:09.679715 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdbf3648-1abd-4d7b-959d-637a1e564adf-kube-api-access-gkzx6\" (UniqueName: \"kubernetes.io/projected/cdbf3648-1abd-4d7b-959d-637a1e564adf-kube-api-access-gkzx6\") pod \"cdbf3648-1abd-4d7b-959d-637a1e564adf\" (UID: \"cdbf3648-1abd-4d7b-959d-637a1e564adf\") " Oct 31 00:47:09.680764 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:09.679755 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdbf3648-1abd-4d7b-959d-637a1e564adf-tls\" (UniqueName: \"kubernetes.io/secret/cdbf3648-1abd-4d7b-959d-637a1e564adf-tls\") pod \"cdbf3648-1abd-4d7b-959d-637a1e564adf\" (UID: \"cdbf3648-1abd-4d7b-959d-637a1e564adf\") " Oct 31 00:47:09.680764 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:09.679809 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/644488ea-9480-4d33-9e58-701b46b95a13-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/644488ea-9480-4d33-9e58-701b46b95a13-prometheus-ca\") pod \"644488ea-9480-4d33-9e58-701b46b95a13\" (UID: \"644488ea-9480-4d33-9e58-701b46b95a13\") " Oct 31 00:47:09.680764 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:09.680385 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/644488ea-9480-4d33-9e58-701b46b95a13-prometheus-ca" pod "644488ea-9480-4d33-9e58-701b46b95a13" (UID: "644488ea-9480-4d33-9e58-701b46b95a13"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:09.681249 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:09.681111 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdbf3648-1abd-4d7b-959d-637a1e564adf-config" pod "cdbf3648-1abd-4d7b-959d-637a1e564adf" (UID: "cdbf3648-1abd-4d7b-959d-637a1e564adf"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:09.683632 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:09.683608 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/644488ea-9480-4d33-9e58-701b46b95a13-prometheus-client-cert" pod "644488ea-9480-4d33-9e58-701b46b95a13" (UID: "644488ea-9480-4d33-9e58-701b46b95a13"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:09.684812 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:09.684776 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdbf3648-1abd-4d7b-959d-637a1e564adf-kube-api-access-gkzx6" pod "cdbf3648-1abd-4d7b-959d-637a1e564adf" (UID: "cdbf3648-1abd-4d7b-959d-637a1e564adf"). InnerVolumeSpecName "kube-api-access-gkzx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:09.684927 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:09.684864 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/644488ea-9480-4d33-9e58-701b46b95a13-tls" pod "644488ea-9480-4d33-9e58-701b46b95a13" (UID: "644488ea-9480-4d33-9e58-701b46b95a13"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:09.686978 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:09.686345 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdbf3648-1abd-4d7b-959d-637a1e564adf-tls" pod "cdbf3648-1abd-4d7b-959d-637a1e564adf" (UID: "cdbf3648-1abd-4d7b-959d-637a1e564adf"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:09.686978 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:09.686342 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/644488ea-9480-4d33-9e58-701b46b95a13-kube-api-access-66djb" pod "644488ea-9480-4d33-9e58-701b46b95a13" (UID: "644488ea-9480-4d33-9e58-701b46b95a13"). InnerVolumeSpecName "kube-api-access-66djb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:09.687444 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:09.687412 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/644488ea-9480-4d33-9e58-701b46b95a13-kube-rbac-proxy-config" pod "644488ea-9480-4d33-9e58-701b46b95a13" (UID: "644488ea-9480-4d33-9e58-701b46b95a13"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:09.781263 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:09.781235 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/644488ea-9480-4d33-9e58-701b46b95a13-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:09.781263 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:09.781263 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkzx6\" (UniqueName: \"kubernetes.io/projected/cdbf3648-1abd-4d7b-959d-637a1e564adf-kube-api-access-gkzx6\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:09.781432 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:09.781274 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/cdbf3648-1abd-4d7b-959d-637a1e564adf-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:09.781432 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:09.781283 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/644488ea-9480-4d33-9e58-701b46b95a13-prometheus-ca\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:09.781432 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:09.781293 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/644488ea-9480-4d33-9e58-701b46b95a13-prometheus-client-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:09.781432 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:09.781302 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66djb\" (UniqueName: \"kubernetes.io/projected/644488ea-9480-4d33-9e58-701b46b95a13-kube-api-access-66djb\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:09.781432 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:09.781311 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cdbf3648-1abd-4d7b-959d-637a1e564adf-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:09.781432 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:09.781319 5274 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/644488ea-9480-4d33-9e58-701b46b95a13-kube-rbac-proxy-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.156182 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.156147 5274 generic.go:356] "Generic (PLEG): container finished" podID="67ff5c67-5207-49be-93f2-81429cd1d8a9" containerID="1a756d003be069795fbca067b851999389e0a68a198e7e0dcd7521ffde128941" exitCode=0 Oct 31 00:47:10.156335 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.156228 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"67ff5c67-5207-49be-93f2-81429cd1d8a9","Type":"ContainerDied","Data":"1a756d003be069795fbca067b851999389e0a68a198e7e0dcd7521ffde128941"} Oct 31 00:47:10.159200 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.159171 5274 generic.go:356] "Generic (PLEG): container finished" podID="434094cd-527e-4be5-bd9d-0a49be98a7bf" containerID="609efa17c1c4aa2ea92704a81291fef031aa600182cd27375f5c47da88de1336" exitCode=0 Oct 31 00:47:10.159200 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.159199 5274 generic.go:356] "Generic (PLEG): container finished" podID="434094cd-527e-4be5-bd9d-0a49be98a7bf" containerID="a6375fe434542308f81abc9825ac4af4f42e514f825cabcd801ec18fe63cbaba" exitCode=0 Oct 31 00:47:10.159200 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.159208 5274 generic.go:356] "Generic (PLEG): container finished" podID="434094cd-527e-4be5-bd9d-0a49be98a7bf" containerID="16ae09a3d99b6c737b11927ca8ebfa6e81ef6c2467de0f43c4adb57bd3782694" exitCode=0 Oct 31 00:47:10.159488 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.159320 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"434094cd-527e-4be5-bd9d-0a49be98a7bf","Type":"ContainerDied","Data":"609efa17c1c4aa2ea92704a81291fef031aa600182cd27375f5c47da88de1336"} Oct 31 00:47:10.159488 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.159378 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"434094cd-527e-4be5-bd9d-0a49be98a7bf","Type":"ContainerDied","Data":"a6375fe434542308f81abc9825ac4af4f42e514f825cabcd801ec18fe63cbaba"} Oct 31 00:47:10.159488 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.159418 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"434094cd-527e-4be5-bd9d-0a49be98a7bf","Type":"ContainerDied","Data":"16ae09a3d99b6c737b11927ca8ebfa6e81ef6c2467de0f43c4adb57bd3782694"} Oct 31 00:47:10.160947 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.160766 5274 generic.go:356] "Generic (PLEG): container finished" podID="cdbf3648-1abd-4d7b-959d-637a1e564adf" containerID="54f456929e1f583631a2e24f97d945cbc62ca6471233d20880b808022bc19e30" exitCode=0 Oct 31 00:47:10.160947 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.160937 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-n684d" event={"ID":"cdbf3648-1abd-4d7b-959d-637a1e564adf","Type":"ContainerDied","Data":"54f456929e1f583631a2e24f97d945cbc62ca6471233d20880b808022bc19e30"} Oct 31 00:47:10.161129 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.161054 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-n684d" event={"ID":"cdbf3648-1abd-4d7b-959d-637a1e564adf","Type":"ContainerDied","Data":"ace4ca5dced3d0227bbf1f63a7e549830dda43e05a6e02952b927d94fcac4aef"} Oct 31 00:47:10.161182 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.161136 5274 scope.go:122] "RemoveContainer" containerID="54f456929e1f583631a2e24f97d945cbc62ca6471233d20880b808022bc19e30" Oct 31 00:47:10.161327 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.161313 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-n684d" Oct 31 00:47:10.164833 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.164312 5274 generic.go:356] "Generic (PLEG): container finished" podID="644488ea-9480-4d33-9e58-701b46b95a13" containerID="23a18d03209ee6ee91728c4e12eb7e7224323ee51b16c3955a3ef61fad2a27a9" exitCode=0 Oct 31 00:47:10.164833 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.164387 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-mpkxh" event={"ID":"644488ea-9480-4d33-9e58-701b46b95a13","Type":"ContainerDied","Data":"23a18d03209ee6ee91728c4e12eb7e7224323ee51b16c3955a3ef61fad2a27a9"} Oct 31 00:47:10.164833 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.164422 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-mpkxh" event={"ID":"644488ea-9480-4d33-9e58-701b46b95a13","Type":"ContainerDied","Data":"eaf6a0d0182765b7e3fa1780d42daeeb82f76a7205d1f78b3e21a0681c35dd5b"} Oct 31 00:47:10.164833 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.164498 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-mpkxh" Oct 31 00:47:10.167656 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.167522 5274 generic.go:356] "Generic (PLEG): container finished" podID="3f69d683-62af-468f-8ae5-f2b2d101bf61" containerID="eb445f800eb5fda3011cc89baf7988dce7a671257953e71fa9453edb67b6b314" exitCode=0 Oct 31 00:47:10.167656 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.167597 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"3f69d683-62af-468f-8ae5-f2b2d101bf61","Type":"ContainerDied","Data":"eb445f800eb5fda3011cc89baf7988dce7a671257953e71fa9453edb67b6b314"} Oct 31 00:47:10.175102 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.175083 5274 scope.go:122] "RemoveContainer" containerID="54f456929e1f583631a2e24f97d945cbc62ca6471233d20880b808022bc19e30" Oct 31 00:47:10.175404 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:47:10.175381 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54f456929e1f583631a2e24f97d945cbc62ca6471233d20880b808022bc19e30\": container with ID starting with 54f456929e1f583631a2e24f97d945cbc62ca6471233d20880b808022bc19e30 not found: ID does not exist" containerID="54f456929e1f583631a2e24f97d945cbc62ca6471233d20880b808022bc19e30" Oct 31 00:47:10.175486 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.175413 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54f456929e1f583631a2e24f97d945cbc62ca6471233d20880b808022bc19e30"} err="failed to get container status \"54f456929e1f583631a2e24f97d945cbc62ca6471233d20880b808022bc19e30\": rpc error: code = NotFound desc = could not find container \"54f456929e1f583631a2e24f97d945cbc62ca6471233d20880b808022bc19e30\": container with ID starting with 54f456929e1f583631a2e24f97d945cbc62ca6471233d20880b808022bc19e30 not found: ID does not exist" Oct 31 00:47:10.175486 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.175438 5274 scope.go:122] "RemoveContainer" containerID="23a18d03209ee6ee91728c4e12eb7e7224323ee51b16c3955a3ef61fad2a27a9" Oct 31 00:47:10.185545 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.185526 5274 scope.go:122] "RemoveContainer" containerID="23a18d03209ee6ee91728c4e12eb7e7224323ee51b16c3955a3ef61fad2a27a9" Oct 31 00:47:10.185814 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:47:10.185796 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23a18d03209ee6ee91728c4e12eb7e7224323ee51b16c3955a3ef61fad2a27a9\": container with ID starting with 23a18d03209ee6ee91728c4e12eb7e7224323ee51b16c3955a3ef61fad2a27a9 not found: ID does not exist" containerID="23a18d03209ee6ee91728c4e12eb7e7224323ee51b16c3955a3ef61fad2a27a9" Oct 31 00:47:10.185885 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.185823 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23a18d03209ee6ee91728c4e12eb7e7224323ee51b16c3955a3ef61fad2a27a9"} err="failed to get container status \"23a18d03209ee6ee91728c4e12eb7e7224323ee51b16c3955a3ef61fad2a27a9\": rpc error: code = NotFound desc = could not find container \"23a18d03209ee6ee91728c4e12eb7e7224323ee51b16c3955a3ef61fad2a27a9\": container with ID starting with 23a18d03209ee6ee91728c4e12eb7e7224323ee51b16c3955a3ef61fad2a27a9 not found: ID does not exist" Oct 31 00:47:10.209653 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.209631 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-n684d"] Oct 31 00:47:10.215304 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.215280 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-n684d"] Oct 31 00:47:10.223915 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.223894 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:10.226920 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.226897 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-mpkxh"] Oct 31 00:47:10.233880 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.233859 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-mpkxh"] Oct 31 00:47:10.271216 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.271195 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:10.284794 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.284774 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"434094cd-527e-4be5-bd9d-0a49be98a7bf\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " Oct 31 00:47:10.284889 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.284802 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-thanos-prometheus-http-client-file\") pod \"434094cd-527e-4be5-bd9d-0a49be98a7bf\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " Oct 31 00:47:10.284889 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.284825 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67ff5c67-5207-49be-93f2-81429cd1d8a9-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/67ff5c67-5207-49be-93f2-81429cd1d8a9-monitoring-tls-certs\") pod \"67ff5c67-5207-49be-93f2-81429cd1d8a9\" (UID: \"67ff5c67-5207-49be-93f2-81429cd1d8a9\") " Oct 31 00:47:10.284889 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.284842 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/434094cd-527e-4be5-bd9d-0a49be98a7bf-tls-assets\" (UniqueName: \"kubernetes.io/projected/434094cd-527e-4be5-bd9d-0a49be98a7bf-tls-assets\") pod \"434094cd-527e-4be5-bd9d-0a49be98a7bf\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " Oct 31 00:47:10.284889 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.284873 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/434094cd-527e-4be5-bd9d-0a49be98a7bf-config-out\" (UniqueName: \"kubernetes.io/empty-dir/434094cd-527e-4be5-bd9d-0a49be98a7bf-config-out\") pod \"434094cd-527e-4be5-bd9d-0a49be98a7bf\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " Oct 31 00:47:10.285098 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.285065 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67ff5c67-5207-49be-93f2-81429cd1d8a9-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/67ff5c67-5207-49be-93f2-81429cd1d8a9-data-science-collector-ta-client-cert\") pod \"67ff5c67-5207-49be-93f2-81429cd1d8a9\" (UID: \"67ff5c67-5207-49be-93f2-81429cd1d8a9\") " Oct 31 00:47:10.285172 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.285113 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/434094cd-527e-4be5-bd9d-0a49be98a7bf-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/434094cd-527e-4be5-bd9d-0a49be98a7bf-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"434094cd-527e-4be5-bd9d-0a49be98a7bf\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " Oct 31 00:47:10.285172 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.285143 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67ff5c67-5207-49be-93f2-81429cd1d8a9-kube-api-access-8hdgz\" (UniqueName: \"kubernetes.io/projected/67ff5c67-5207-49be-93f2-81429cd1d8a9-kube-api-access-8hdgz\") pod \"67ff5c67-5207-49be-93f2-81429cd1d8a9\" (UID: \"67ff5c67-5207-49be-93f2-81429cd1d8a9\") " Oct 31 00:47:10.285292 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.285195 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/434094cd-527e-4be5-bd9d-0a49be98a7bf-kube-api-access-x6tnj\" (UniqueName: \"kubernetes.io/projected/434094cd-527e-4be5-bd9d-0a49be98a7bf-kube-api-access-x6tnj\") pod \"434094cd-527e-4be5-bd9d-0a49be98a7bf\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " Oct 31 00:47:10.285292 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.285245 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/434094cd-527e-4be5-bd9d-0a49be98a7bf-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/434094cd-527e-4be5-bd9d-0a49be98a7bf-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"434094cd-527e-4be5-bd9d-0a49be98a7bf\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " Oct 31 00:47:10.285292 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.285273 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"434094cd-527e-4be5-bd9d-0a49be98a7bf\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " Oct 31 00:47:10.285443 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.285329 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/434094cd-527e-4be5-bd9d-0a49be98a7bf-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/434094cd-527e-4be5-bd9d-0a49be98a7bf-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"434094cd-527e-4be5-bd9d-0a49be98a7bf\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " Oct 31 00:47:10.285443 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.285360 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67ff5c67-5207-49be-93f2-81429cd1d8a9-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/67ff5c67-5207-49be-93f2-81429cd1d8a9-otlp-tls-certs\") pod \"67ff5c67-5207-49be-93f2-81429cd1d8a9\" (UID: \"67ff5c67-5207-49be-93f2-81429cd1d8a9\") " Oct 31 00:47:10.285443 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.285395 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-web-config\" (UniqueName: \"kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-web-config\") pod \"434094cd-527e-4be5-bd9d-0a49be98a7bf\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " Oct 31 00:47:10.285443 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.285431 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67ff5c67-5207-49be-93f2-81429cd1d8a9-otc-internal\" (UniqueName: \"kubernetes.io/configmap/67ff5c67-5207-49be-93f2-81429cd1d8a9-otc-internal\") pod \"67ff5c67-5207-49be-93f2-81429cd1d8a9\" (UID: \"67ff5c67-5207-49be-93f2-81429cd1d8a9\") " Oct 31 00:47:10.285669 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.285515 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"434094cd-527e-4be5-bd9d-0a49be98a7bf\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " Oct 31 00:47:10.285669 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.285570 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-secret-prometheus-web-tls-ca\") pod \"434094cd-527e-4be5-bd9d-0a49be98a7bf\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " Oct 31 00:47:10.285669 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.285597 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67ff5c67-5207-49be-93f2-81429cd1d8a9-tls-certs\" (UniqueName: \"kubernetes.io/secret/67ff5c67-5207-49be-93f2-81429cd1d8a9-tls-certs\") pod \"67ff5c67-5207-49be-93f2-81429cd1d8a9\" (UID: \"67ff5c67-5207-49be-93f2-81429cd1d8a9\") " Oct 31 00:47:10.285669 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.285637 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-config\" (UniqueName: \"kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-config\") pod \"434094cd-527e-4be5-bd9d-0a49be98a7bf\" (UID: \"434094cd-527e-4be5-bd9d-0a49be98a7bf\") " Oct 31 00:47:10.287299 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.286356 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/434094cd-527e-4be5-bd9d-0a49be98a7bf-prometheus-data-science-monitoringstack-rulefiles-2" pod "434094cd-527e-4be5-bd9d-0a49be98a7bf" (UID: "434094cd-527e-4be5-bd9d-0a49be98a7bf"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:10.287299 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.286799 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/434094cd-527e-4be5-bd9d-0a49be98a7bf-prometheus-data-science-monitoringstack-rulefiles-0" pod "434094cd-527e-4be5-bd9d-0a49be98a7bf" (UID: "434094cd-527e-4be5-bd9d-0a49be98a7bf"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:10.287299 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.287005 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67ff5c67-5207-49be-93f2-81429cd1d8a9-otc-internal" pod "67ff5c67-5207-49be-93f2-81429cd1d8a9" (UID: "67ff5c67-5207-49be-93f2-81429cd1d8a9"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:10.287299 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.287258 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/434094cd-527e-4be5-bd9d-0a49be98a7bf-prometheus-data-science-monitoringstack-rulefiles-1" pod "434094cd-527e-4be5-bd9d-0a49be98a7bf" (UID: "434094cd-527e-4be5-bd9d-0a49be98a7bf"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:10.287679 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.287644 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/434094cd-527e-4be5-bd9d-0a49be98a7bf-config-out" pod "434094cd-527e-4be5-bd9d-0a49be98a7bf" (UID: "434094cd-527e-4be5-bd9d-0a49be98a7bf"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:10.288652 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.288605 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67ff5c67-5207-49be-93f2-81429cd1d8a9-data-science-collector-ta-client-cert" pod "67ff5c67-5207-49be-93f2-81429cd1d8a9" (UID: "67ff5c67-5207-49be-93f2-81429cd1d8a9"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:10.288652 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.288611 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/434094cd-527e-4be5-bd9d-0a49be98a7bf-tls-assets" pod "434094cd-527e-4be5-bd9d-0a49be98a7bf" (UID: "434094cd-527e-4be5-bd9d-0a49be98a7bf"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:10.289377 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.289232 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-thanos-prometheus-http-client-file" pod "434094cd-527e-4be5-bd9d-0a49be98a7bf" (UID: "434094cd-527e-4be5-bd9d-0a49be98a7bf"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:10.289377 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.289303 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "434094cd-527e-4be5-bd9d-0a49be98a7bf" (UID: "434094cd-527e-4be5-bd9d-0a49be98a7bf"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:10.289377 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.289322 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-secret-prometheus-web-tls-ca" pod "434094cd-527e-4be5-bd9d-0a49be98a7bf" (UID: "434094cd-527e-4be5-bd9d-0a49be98a7bf"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:10.289377 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.289355 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67ff5c67-5207-49be-93f2-81429cd1d8a9-otlp-tls-certs" pod "67ff5c67-5207-49be-93f2-81429cd1d8a9" (UID: "67ff5c67-5207-49be-93f2-81429cd1d8a9"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:10.289377 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.289367 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "434094cd-527e-4be5-bd9d-0a49be98a7bf" (UID: "434094cd-527e-4be5-bd9d-0a49be98a7bf"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:10.289642 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.289526 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67ff5c67-5207-49be-93f2-81429cd1d8a9-monitoring-tls-certs" pod "67ff5c67-5207-49be-93f2-81429cd1d8a9" (UID: "67ff5c67-5207-49be-93f2-81429cd1d8a9"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:10.289642 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.289563 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-config" pod "434094cd-527e-4be5-bd9d-0a49be98a7bf" (UID: "434094cd-527e-4be5-bd9d-0a49be98a7bf"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:10.289642 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.289626 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/434094cd-527e-4be5-bd9d-0a49be98a7bf-kube-api-access-x6tnj" pod "434094cd-527e-4be5-bd9d-0a49be98a7bf" (UID: "434094cd-527e-4be5-bd9d-0a49be98a7bf"). InnerVolumeSpecName "kube-api-access-x6tnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:10.291070 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.291039 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67ff5c67-5207-49be-93f2-81429cd1d8a9-kube-api-access-8hdgz" pod "67ff5c67-5207-49be-93f2-81429cd1d8a9" (UID: "67ff5c67-5207-49be-93f2-81429cd1d8a9"). InnerVolumeSpecName "kube-api-access-8hdgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:10.291165 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.291120 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67ff5c67-5207-49be-93f2-81429cd1d8a9-tls-certs" pod "67ff5c67-5207-49be-93f2-81429cd1d8a9" (UID: "67ff5c67-5207-49be-93f2-81429cd1d8a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:10.292677 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.292661 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a" pod "434094cd-527e-4be5-bd9d-0a49be98a7bf" (UID: "434094cd-527e-4be5-bd9d-0a49be98a7bf"). InnerVolumeSpecName "pvc-63e34709-8d85-4142-b53a-f655698606ff". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:47:10.302094 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.302070 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-web-config" pod "434094cd-527e-4be5-bd9d-0a49be98a7bf" (UID: "434094cd-527e-4be5-bd9d-0a49be98a7bf"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:10.387069 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.387043 5274 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/434094cd-527e-4be5-bd9d-0a49be98a7bf-config-out\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.387069 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.387066 5274 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/67ff5c67-5207-49be-93f2-81429cd1d8a9-data-science-collector-ta-client-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.387245 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.387077 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/434094cd-527e-4be5-bd9d-0a49be98a7bf-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.387245 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.387087 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hdgz\" (UniqueName: \"kubernetes.io/projected/67ff5c67-5207-49be-93f2-81429cd1d8a9-kube-api-access-8hdgz\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.387245 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.387097 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6tnj\" (UniqueName: \"kubernetes.io/projected/434094cd-527e-4be5-bd9d-0a49be98a7bf-kube-api-access-x6tnj\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.387245 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.387106 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/434094cd-527e-4be5-bd9d-0a49be98a7bf-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.387245 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.387115 5274 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.387245 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.387125 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/434094cd-527e-4be5-bd9d-0a49be98a7bf-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.387245 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.387133 5274 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/67ff5c67-5207-49be-93f2-81429cd1d8a9-otlp-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.387245 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.387143 5274 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-web-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.387245 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.387151 5274 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/67ff5c67-5207-49be-93f2-81429cd1d8a9-otc-internal\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.387245 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.387177 5274 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" " Oct 31 00:47:10.387245 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.387187 5274 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-secret-prometheus-web-tls-ca\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.387245 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.387200 5274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67ff5c67-5207-49be-93f2-81429cd1d8a9-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.387245 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.387212 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.387245 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.387221 5274 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.387245 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.387244 5274 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/434094cd-527e-4be5-bd9d-0a49be98a7bf-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.387245 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.387252 5274 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/67ff5c67-5207-49be-93f2-81429cd1d8a9-monitoring-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.387686 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.387260 5274 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/434094cd-527e-4be5-bd9d-0a49be98a7bf-tls-assets\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.398236 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.398215 5274 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-63e34709-8d85-4142-b53a-f655698606ff" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a") on node "ip-10-0-129-70.us-east-2.compute.internal" Oct 31 00:47:10.488152 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.488126 5274 reconciler_common.go:299] "Volume detached for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"csi-3beb91b3ebf177571ca474589da3560cd18787cfa50b2a0b9bfef85661e6cf21\"" Oct 31 00:47:10.734807 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:10.734769 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="3f69d683-62af-468f-8ae5-f2b2d101bf61" containerName="alertmanager" probeResult="failure" output="Get \"http://10.134.0.119:9093/-/ready\": dial tcp 10.134.0.119:9093: connect: connection refused" Oct 31 00:47:11.175753 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:11.175726 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:11.175974 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:11.175724 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"67ff5c67-5207-49be-93f2-81429cd1d8a9","Type":"ContainerDied","Data":"cc4a318ee664d7356a0a958033121a444484aa2a3dd20f358e2a21d5f6acc79b"} Oct 31 00:47:11.175974 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:11.175873 5274 scope.go:122] "RemoveContainer" containerID="1a756d003be069795fbca067b851999389e0a68a198e7e0dcd7521ffde128941" Oct 31 00:47:11.178416 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:11.178394 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"434094cd-527e-4be5-bd9d-0a49be98a7bf","Type":"ContainerDied","Data":"1a9459be232d74b0cfb54f87d01f8987b01697af28cf05cb70d7594b92a7d2bc"} Oct 31 00:47:11.178520 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:11.178455 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:11.184576 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:11.184559 5274 scope.go:122] "RemoveContainer" containerID="609efa17c1c4aa2ea92704a81291fef031aa600182cd27375f5c47da88de1336" Oct 31 00:47:11.192464 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:11.192449 5274 scope.go:122] "RemoveContainer" containerID="a6375fe434542308f81abc9825ac4af4f42e514f825cabcd801ec18fe63cbaba" Oct 31 00:47:11.201636 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:11.201609 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:47:11.205240 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:11.205221 5274 scope.go:122] "RemoveContainer" containerID="16ae09a3d99b6c737b11927ca8ebfa6e81ef6c2467de0f43c4adb57bd3782694" Oct 31 00:47:11.206662 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:11.206645 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:47:11.213356 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:11.213337 5274 scope.go:122] "RemoveContainer" containerID="8e33114c22bf125e2825dd32ed66068d72bc92e69e7e5c09b5bac4066622219d" Oct 31 00:47:11.221716 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:11.221689 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:47:11.225851 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:11.225831 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:47:11.481346 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:11.481274 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="434094cd-527e-4be5-bd9d-0a49be98a7bf" path="/var/lib/kubelet/pods/434094cd-527e-4be5-bd9d-0a49be98a7bf/volumes" Oct 31 00:47:11.481786 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:11.481773 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="644488ea-9480-4d33-9e58-701b46b95a13" path="/var/lib/kubelet/pods/644488ea-9480-4d33-9e58-701b46b95a13/volumes" Oct 31 00:47:11.482426 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:11.482411 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67ff5c67-5207-49be-93f2-81429cd1d8a9" path="/var/lib/kubelet/pods/67ff5c67-5207-49be-93f2-81429cd1d8a9/volumes" Oct 31 00:47:11.482766 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:11.482754 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdbf3648-1abd-4d7b-959d-637a1e564adf" path="/var/lib/kubelet/pods/cdbf3648-1abd-4d7b-959d-637a1e564adf/volumes" Oct 31 00:47:15.735043 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:15.735005 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="3f69d683-62af-468f-8ae5-f2b2d101bf61" containerName="alertmanager" probeResult="failure" output="Get \"http://10.134.0.119:9093/-/ready\": dial tcp 10.134.0.119:9093: connect: connection refused" Oct 31 00:47:17.688267 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:17.688239 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-lffkt"] Oct 31 00:47:17.689020 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:17.688998 5274 cpu_manager.go:414] "containerMap: removing container" podUID="434094cd-527e-4be5-bd9d-0a49be98a7bf" containerName="prometheus" Oct 31 00:47:17.689020 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:17.689018 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="434094cd-527e-4be5-bd9d-0a49be98a7bf" containerName="prometheus" Oct 31 00:47:17.689162 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:17.689029 5274 cpu_manager.go:414] "containerMap: removing container" podUID="644488ea-9480-4d33-9e58-701b46b95a13" containerName="kube-rbac-proxy" Oct 31 00:47:17.689162 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:17.689035 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="644488ea-9480-4d33-9e58-701b46b95a13" containerName="kube-rbac-proxy" Oct 31 00:47:17.689162 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:17.689050 5274 cpu_manager.go:414] "containerMap: removing container" podUID="cdbf3648-1abd-4d7b-959d-637a1e564adf" containerName="korrel8r" Oct 31 00:47:17.689162 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:17.689055 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdbf3648-1abd-4d7b-959d-637a1e564adf" containerName="korrel8r" Oct 31 00:47:17.689162 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:17.689061 5274 cpu_manager.go:414] "containerMap: removing container" podUID="434094cd-527e-4be5-bd9d-0a49be98a7bf" containerName="thanos-sidecar" Oct 31 00:47:17.689162 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:17.689066 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="434094cd-527e-4be5-bd9d-0a49be98a7bf" containerName="thanos-sidecar" Oct 31 00:47:17.689162 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:17.689077 5274 cpu_manager.go:414] "containerMap: removing container" podUID="434094cd-527e-4be5-bd9d-0a49be98a7bf" containerName="config-reloader" Oct 31 00:47:17.689162 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:17.689081 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="434094cd-527e-4be5-bd9d-0a49be98a7bf" containerName="config-reloader" Oct 31 00:47:17.689162 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:17.689090 5274 cpu_manager.go:414] "containerMap: removing container" podUID="67ff5c67-5207-49be-93f2-81429cd1d8a9" containerName="otc-container" Oct 31 00:47:17.689162 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:17.689094 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67ff5c67-5207-49be-93f2-81429cd1d8a9" containerName="otc-container" Oct 31 00:47:17.689162 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:17.689104 5274 cpu_manager.go:414] "containerMap: removing container" podUID="434094cd-527e-4be5-bd9d-0a49be98a7bf" containerName="init-config-reloader" Oct 31 00:47:17.689162 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:17.689110 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="434094cd-527e-4be5-bd9d-0a49be98a7bf" containerName="init-config-reloader" Oct 31 00:47:17.689162 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:17.689159 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="434094cd-527e-4be5-bd9d-0a49be98a7bf" containerName="config-reloader" Oct 31 00:47:17.689536 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:17.689167 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="67ff5c67-5207-49be-93f2-81429cd1d8a9" containerName="otc-container" Oct 31 00:47:17.689536 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:17.689177 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="434094cd-527e-4be5-bd9d-0a49be98a7bf" containerName="prometheus" Oct 31 00:47:17.689536 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:17.689183 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="644488ea-9480-4d33-9e58-701b46b95a13" containerName="kube-rbac-proxy" Oct 31 00:47:17.689536 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:17.689190 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdbf3648-1abd-4d7b-959d-637a1e564adf" containerName="korrel8r" Oct 31 00:47:17.689536 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:17.689194 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="434094cd-527e-4be5-bd9d-0a49be98a7bf" containerName="thanos-sidecar" Oct 31 00:47:17.693445 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:17.693428 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lffkt" Oct 31 00:47:17.700771 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:17.700746 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-lffkt"] Oct 31 00:47:17.715351 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:17.715328 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 31 00:47:17.715481 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:17.715434 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-fpj6t\"" Oct 31 00:47:17.722141 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:17.722125 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 31 00:47:17.746191 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:17.746168 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn64v\" (UniqueName: \"kubernetes.io/projected/46200120-336e-49c2-81f2-fc7a540ca5df-kube-api-access-pn64v\") pod \"korrel8r-5cbcc89b8f-lffkt\" (UID: \"46200120-336e-49c2-81f2-fc7a540ca5df\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lffkt" Oct 31 00:47:17.746296 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:17.746219 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/46200120-336e-49c2-81f2-fc7a540ca5df-config\") pod \"korrel8r-5cbcc89b8f-lffkt\" (UID: \"46200120-336e-49c2-81f2-fc7a540ca5df\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lffkt" Oct 31 00:47:17.746376 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:17.746359 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/46200120-336e-49c2-81f2-fc7a540ca5df-tls\") pod \"korrel8r-5cbcc89b8f-lffkt\" (UID: \"46200120-336e-49c2-81f2-fc7a540ca5df\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lffkt" Oct 31 00:47:17.847608 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:17.847581 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/46200120-336e-49c2-81f2-fc7a540ca5df-config\") pod \"korrel8r-5cbcc89b8f-lffkt\" (UID: \"46200120-336e-49c2-81f2-fc7a540ca5df\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lffkt" Oct 31 00:47:17.847746 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:17.847625 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/46200120-336e-49c2-81f2-fc7a540ca5df-tls\") pod \"korrel8r-5cbcc89b8f-lffkt\" (UID: \"46200120-336e-49c2-81f2-fc7a540ca5df\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lffkt" Oct 31 00:47:17.847746 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:17.847670 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn64v\" (UniqueName: \"kubernetes.io/projected/46200120-336e-49c2-81f2-fc7a540ca5df-kube-api-access-pn64v\") pod \"korrel8r-5cbcc89b8f-lffkt\" (UID: \"46200120-336e-49c2-81f2-fc7a540ca5df\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lffkt" Oct 31 00:47:17.847895 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:47:17.847788 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/korrel8r-tls: secret "korrel8r-tls" not found Oct 31 00:47:17.847895 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:47:17.847850 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46200120-336e-49c2-81f2-fc7a540ca5df-tls podName:46200120-336e-49c2-81f2-fc7a540ca5df nodeName:}" failed. No retries permitted until 2026-10-31 00:47:18.347832346 +0000 UTC m=+2874.984651052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls" (UniqueName: "kubernetes.io/secret/46200120-336e-49c2-81f2-fc7a540ca5df-tls") pod "korrel8r-5cbcc89b8f-lffkt" (UID: "46200120-336e-49c2-81f2-fc7a540ca5df") : secret "korrel8r-tls" not found Oct 31 00:47:17.848561 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:17.848527 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/46200120-336e-49c2-81f2-fc7a540ca5df-config\") pod \"korrel8r-5cbcc89b8f-lffkt\" (UID: \"46200120-336e-49c2-81f2-fc7a540ca5df\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lffkt" Oct 31 00:47:17.867155 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:17.867134 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn64v\" (UniqueName: \"kubernetes.io/projected/46200120-336e-49c2-81f2-fc7a540ca5df-kube-api-access-pn64v\") pod \"korrel8r-5cbcc89b8f-lffkt\" (UID: \"46200120-336e-49c2-81f2-fc7a540ca5df\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lffkt" Oct 31 00:47:18.141346 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:18.141315 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:47:18.144844 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:18.144827 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:18.149688 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:18.149646 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:47:18.150013 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:18.149996 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:47:18.150155 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:18.150058 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:47:18.152432 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:18.152414 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:47:18.152626 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:18.152610 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:47:18.161850 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:18.161828 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:47:18.250820 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:18.250788 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb305977-92f9-45c8-9f97-e37577330b72-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"fb305977-92f9-45c8-9f97-e37577330b72\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:18.250989 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:18.250838 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb305977-92f9-45c8-9f97-e37577330b72-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"fb305977-92f9-45c8-9f97-e37577330b72\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:18.250989 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:18.250868 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fb305977-92f9-45c8-9f97-e37577330b72-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"fb305977-92f9-45c8-9f97-e37577330b72\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:18.251092 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:18.251010 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/fb305977-92f9-45c8-9f97-e37577330b72-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"fb305977-92f9-45c8-9f97-e37577330b72\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:18.251131 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:18.251092 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djtsq\" (UniqueName: \"kubernetes.io/projected/fb305977-92f9-45c8-9f97-e37577330b72-kube-api-access-djtsq\") pod \"data-science-collector-collector-0\" (UID: \"fb305977-92f9-45c8-9f97-e37577330b72\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:18.251131 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:18.251123 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb305977-92f9-45c8-9f97-e37577330b72-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"fb305977-92f9-45c8-9f97-e37577330b72\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:18.351711 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:18.351678 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb305977-92f9-45c8-9f97-e37577330b72-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"fb305977-92f9-45c8-9f97-e37577330b72\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:18.351913 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:18.351721 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/46200120-336e-49c2-81f2-fc7a540ca5df-tls\") pod \"korrel8r-5cbcc89b8f-lffkt\" (UID: \"46200120-336e-49c2-81f2-fc7a540ca5df\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lffkt" Oct 31 00:47:18.351913 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:18.351741 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb305977-92f9-45c8-9f97-e37577330b72-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"fb305977-92f9-45c8-9f97-e37577330b72\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:18.351913 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:18.351771 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fb305977-92f9-45c8-9f97-e37577330b72-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"fb305977-92f9-45c8-9f97-e37577330b72\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:18.351913 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:18.351799 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/fb305977-92f9-45c8-9f97-e37577330b72-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"fb305977-92f9-45c8-9f97-e37577330b72\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:18.351913 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:18.351857 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djtsq\" (UniqueName: \"kubernetes.io/projected/fb305977-92f9-45c8-9f97-e37577330b72-kube-api-access-djtsq\") pod \"data-science-collector-collector-0\" (UID: \"fb305977-92f9-45c8-9f97-e37577330b72\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:18.351913 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:18.351884 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb305977-92f9-45c8-9f97-e37577330b72-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"fb305977-92f9-45c8-9f97-e37577330b72\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:18.352427 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:47:18.351995 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found Oct 31 00:47:18.352427 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:47:18.352081 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb305977-92f9-45c8-9f97-e37577330b72-otlp-tls-certs podName:fb305977-92f9-45c8-9f97-e37577330b72 nodeName:}" failed. No retries permitted until 2026-10-31 00:47:18.852059664 +0000 UTC m=+2875.488878357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/fb305977-92f9-45c8-9f97-e37577330b72-otlp-tls-certs") pod "data-science-collector-collector-0" (UID: "fb305977-92f9-45c8-9f97-e37577330b72") : secret "data-science-collector-collector-headless-tls" not found Oct 31 00:47:18.352567 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:18.352519 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fb305977-92f9-45c8-9f97-e37577330b72-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"fb305977-92f9-45c8-9f97-e37577330b72\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:18.354374 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:18.354347 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/46200120-336e-49c2-81f2-fc7a540ca5df-tls\") pod \"korrel8r-5cbcc89b8f-lffkt\" (UID: \"46200120-336e-49c2-81f2-fc7a540ca5df\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lffkt" Oct 31 00:47:18.354476 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:18.354353 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb305977-92f9-45c8-9f97-e37577330b72-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"fb305977-92f9-45c8-9f97-e37577330b72\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:18.354684 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:18.354661 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb305977-92f9-45c8-9f97-e37577330b72-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"fb305977-92f9-45c8-9f97-e37577330b72\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:18.354757 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:18.354682 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/fb305977-92f9-45c8-9f97-e37577330b72-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"fb305977-92f9-45c8-9f97-e37577330b72\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:18.365083 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:18.365062 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djtsq\" (UniqueName: \"kubernetes.io/projected/fb305977-92f9-45c8-9f97-e37577330b72-kube-api-access-djtsq\") pod \"data-science-collector-collector-0\" (UID: \"fb305977-92f9-45c8-9f97-e37577330b72\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:18.607631 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:18.607565 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lffkt" Oct 31 00:47:18.856320 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:18.856274 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb305977-92f9-45c8-9f97-e37577330b72-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"fb305977-92f9-45c8-9f97-e37577330b72\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:18.858757 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:18.858710 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb305977-92f9-45c8-9f97-e37577330b72-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"fb305977-92f9-45c8-9f97-e37577330b72\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:18.883771 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:18.883748 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-lffkt"] Oct 31 00:47:18.886236 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:47:18.886209 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46200120_336e_49c2_81f2_fc7a540ca5df.slice/crio-b88ed78855953b70ef3062bc9a11870c8e137b1c5be64c468cc3bdc12ae30fb6 WatchSource:0}: Error finding container b88ed78855953b70ef3062bc9a11870c8e137b1c5be64c468cc3bdc12ae30fb6: Status 404 returned error can't find the container with id b88ed78855953b70ef3062bc9a11870c8e137b1c5be64c468cc3bdc12ae30fb6 Oct 31 00:47:19.064325 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:19.064292 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:19.233153 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:19.233123 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lffkt" event={"ID":"46200120-336e-49c2-81f2-fc7a540ca5df","Type":"ContainerStarted","Data":"c081728fea9b28106665289c08489a5b4fea86ec128f6f918ba0a133fcac70ca"} Oct 31 00:47:19.233335 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:19.233179 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lffkt" Oct 31 00:47:19.233335 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:19.233202 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lffkt" event={"ID":"46200120-336e-49c2-81f2-fc7a540ca5df","Type":"ContainerStarted","Data":"b88ed78855953b70ef3062bc9a11870c8e137b1c5be64c468cc3bdc12ae30fb6"} Oct 31 00:47:19.258969 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:19.258916 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lffkt" podStartSLOduration=2.258904287 podStartE2EDuration="2.258904287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:19.257253225 +0000 UTC m=+2875.894071941" watchObservedRunningTime="2026-10-31 00:47:19.258904287 +0000 UTC m=+2875.895723026" Oct 31 00:47:19.350407 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:19.350384 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:47:19.352083 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:47:19.352062 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb305977_92f9_45c8_9f97_e37577330b72.slice/crio-998c7a078574f8fe578ad15b03525e10c25cca661265d3cc4abf72b805b0f3ad WatchSource:0}: Error finding container 998c7a078574f8fe578ad15b03525e10c25cca661265d3cc4abf72b805b0f3ad: Status 404 returned error can't find the container with id 998c7a078574f8fe578ad15b03525e10c25cca661265d3cc4abf72b805b0f3ad Oct 31 00:47:19.354005 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:19.353971 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:47:19.354078 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:19.354052 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:47:19.751880 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:19.751853 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:47:19.864660 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:19.864637 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f69d683-62af-468f-8ae5-f2b2d101bf61-web-config\" (UniqueName: \"kubernetes.io/secret/3f69d683-62af-468f-8ae5-f2b2d101bf61-web-config\") pod \"3f69d683-62af-468f-8ae5-f2b2d101bf61\" (UID: \"3f69d683-62af-468f-8ae5-f2b2d101bf61\") " Oct 31 00:47:19.865088 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:19.864683 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f69d683-62af-468f-8ae5-f2b2d101bf61-config-out\" (UniqueName: \"kubernetes.io/empty-dir/3f69d683-62af-468f-8ae5-f2b2d101bf61-config-out\") pod \"3f69d683-62af-468f-8ae5-f2b2d101bf61\" (UID: \"3f69d683-62af-468f-8ae5-f2b2d101bf61\") " Oct 31 00:47:19.865088 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:19.864732 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f69d683-62af-468f-8ae5-f2b2d101bf61-tls-assets\" (UniqueName: \"kubernetes.io/projected/3f69d683-62af-468f-8ae5-f2b2d101bf61-tls-assets\") pod \"3f69d683-62af-468f-8ae5-f2b2d101bf61\" (UID: \"3f69d683-62af-468f-8ae5-f2b2d101bf61\") " Oct 31 00:47:19.865088 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:19.864764 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f69d683-62af-468f-8ae5-f2b2d101bf61-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/3f69d683-62af-468f-8ae5-f2b2d101bf61-alertmanager-data-science-monitoringstack-db\") pod \"3f69d683-62af-468f-8ae5-f2b2d101bf61\" (UID: \"3f69d683-62af-468f-8ae5-f2b2d101bf61\") " Oct 31 00:47:19.865088 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:19.864808 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f69d683-62af-468f-8ae5-f2b2d101bf61-kube-api-access-r725b\" (UniqueName: \"kubernetes.io/projected/3f69d683-62af-468f-8ae5-f2b2d101bf61-kube-api-access-r725b\") pod \"3f69d683-62af-468f-8ae5-f2b2d101bf61\" (UID: \"3f69d683-62af-468f-8ae5-f2b2d101bf61\") " Oct 31 00:47:19.865088 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:19.864829 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f69d683-62af-468f-8ae5-f2b2d101bf61-config-volume\" (UniqueName: \"kubernetes.io/secret/3f69d683-62af-468f-8ae5-f2b2d101bf61-config-volume\") pod \"3f69d683-62af-468f-8ae5-f2b2d101bf61\" (UID: \"3f69d683-62af-468f-8ae5-f2b2d101bf61\") " Oct 31 00:47:19.865088 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:19.864857 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f69d683-62af-468f-8ae5-f2b2d101bf61-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3f69d683-62af-468f-8ae5-f2b2d101bf61-cluster-tls-config\") pod \"3f69d683-62af-468f-8ae5-f2b2d101bf61\" (UID: \"3f69d683-62af-468f-8ae5-f2b2d101bf61\") " Oct 31 00:47:19.865422 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:19.865179 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f69d683-62af-468f-8ae5-f2b2d101bf61-alertmanager-data-science-monitoringstack-db" pod "3f69d683-62af-468f-8ae5-f2b2d101bf61" (UID: "3f69d683-62af-468f-8ae5-f2b2d101bf61"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:19.867301 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:19.867256 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f69d683-62af-468f-8ae5-f2b2d101bf61-config-volume" pod "3f69d683-62af-468f-8ae5-f2b2d101bf61" (UID: "3f69d683-62af-468f-8ae5-f2b2d101bf61"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:19.867420 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:19.867362 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f69d683-62af-468f-8ae5-f2b2d101bf61-config-out" pod "3f69d683-62af-468f-8ae5-f2b2d101bf61" (UID: "3f69d683-62af-468f-8ae5-f2b2d101bf61"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:19.867497 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:19.867419 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f69d683-62af-468f-8ae5-f2b2d101bf61-tls-assets" pod "3f69d683-62af-468f-8ae5-f2b2d101bf61" (UID: "3f69d683-62af-468f-8ae5-f2b2d101bf61"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:19.867497 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:19.867438 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f69d683-62af-468f-8ae5-f2b2d101bf61-kube-api-access-r725b" pod "3f69d683-62af-468f-8ae5-f2b2d101bf61" (UID: "3f69d683-62af-468f-8ae5-f2b2d101bf61"). InnerVolumeSpecName "kube-api-access-r725b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:19.871263 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:19.871227 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f69d683-62af-468f-8ae5-f2b2d101bf61-cluster-tls-config" pod "3f69d683-62af-468f-8ae5-f2b2d101bf61" (UID: "3f69d683-62af-468f-8ae5-f2b2d101bf61"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:19.878414 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:19.878389 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f69d683-62af-468f-8ae5-f2b2d101bf61-web-config" pod "3f69d683-62af-468f-8ae5-f2b2d101bf61" (UID: "3f69d683-62af-468f-8ae5-f2b2d101bf61"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:19.966371 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:19.966347 5274 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3f69d683-62af-468f-8ae5-f2b2d101bf61-tls-assets\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:19.966371 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:19.966370 5274 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/3f69d683-62af-468f-8ae5-f2b2d101bf61-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:19.966510 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:19.966381 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r725b\" (UniqueName: \"kubernetes.io/projected/3f69d683-62af-468f-8ae5-f2b2d101bf61-kube-api-access-r725b\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:19.966510 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:19.966392 5274 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3f69d683-62af-468f-8ae5-f2b2d101bf61-config-volume\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:19.966510 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:19.966402 5274 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3f69d683-62af-468f-8ae5-f2b2d101bf61-cluster-tls-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:19.966510 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:19.966410 5274 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3f69d683-62af-468f-8ae5-f2b2d101bf61-web-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:19.966510 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:19.966418 5274 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3f69d683-62af-468f-8ae5-f2b2d101bf61-config-out\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:20.240317 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:20.240241 5274 generic.go:356] "Generic (PLEG): container finished" podID="3f69d683-62af-468f-8ae5-f2b2d101bf61" containerID="2273a440ff8b094cfffc936bf77205754ee60a3b4410bfc11ee6d365776f13ab" exitCode=0 Oct 31 00:47:20.240317 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:20.240309 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:47:20.240498 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:20.240325 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"3f69d683-62af-468f-8ae5-f2b2d101bf61","Type":"ContainerDied","Data":"2273a440ff8b094cfffc936bf77205754ee60a3b4410bfc11ee6d365776f13ab"} Oct 31 00:47:20.240498 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:20.240388 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"3f69d683-62af-468f-8ae5-f2b2d101bf61","Type":"ContainerDied","Data":"46d1d1d231eb87eb82b0e555839dff04aadb02d9f8bf766686f9c52642f4dd8c"} Oct 31 00:47:20.240498 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:20.240421 5274 scope.go:122] "RemoveContainer" containerID="eb445f800eb5fda3011cc89baf7988dce7a671257953e71fa9453edb67b6b314" Oct 31 00:47:20.242379 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:20.242348 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"fb305977-92f9-45c8-9f97-e37577330b72","Type":"ContainerStarted","Data":"7a645a5352ceb300e6c997ae1b35e5a347a511c7723d43d4a254964e7e30cece"} Oct 31 00:47:20.242515 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:20.242405 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"fb305977-92f9-45c8-9f97-e37577330b72","Type":"ContainerStarted","Data":"998c7a078574f8fe578ad15b03525e10c25cca661265d3cc4abf72b805b0f3ad"} Oct 31 00:47:20.249495 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:20.249478 5274 scope.go:122] "RemoveContainer" containerID="2273a440ff8b094cfffc936bf77205754ee60a3b4410bfc11ee6d365776f13ab" Oct 31 00:47:20.257029 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:20.257002 5274 scope.go:122] "RemoveContainer" containerID="003e2f64063208af10073e83c74055a6ec87b08bdcf579f15067dc9a92078124" Oct 31 00:47:20.263991 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:20.263950 5274 scope.go:122] "RemoveContainer" containerID="eb445f800eb5fda3011cc89baf7988dce7a671257953e71fa9453edb67b6b314" Oct 31 00:47:20.264255 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:47:20.264226 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb445f800eb5fda3011cc89baf7988dce7a671257953e71fa9453edb67b6b314\": container with ID starting with eb445f800eb5fda3011cc89baf7988dce7a671257953e71fa9453edb67b6b314 not found: ID does not exist" containerID="eb445f800eb5fda3011cc89baf7988dce7a671257953e71fa9453edb67b6b314" Oct 31 00:47:20.264345 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:20.264257 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb445f800eb5fda3011cc89baf7988dce7a671257953e71fa9453edb67b6b314"} err="failed to get container status \"eb445f800eb5fda3011cc89baf7988dce7a671257953e71fa9453edb67b6b314\": rpc error: code = NotFound desc = could not find container \"eb445f800eb5fda3011cc89baf7988dce7a671257953e71fa9453edb67b6b314\": container with ID starting with eb445f800eb5fda3011cc89baf7988dce7a671257953e71fa9453edb67b6b314 not found: ID does not exist" Oct 31 00:47:20.264345 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:20.264278 5274 scope.go:122] "RemoveContainer" containerID="2273a440ff8b094cfffc936bf77205754ee60a3b4410bfc11ee6d365776f13ab" Oct 31 00:47:20.264520 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:47:20.264501 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2273a440ff8b094cfffc936bf77205754ee60a3b4410bfc11ee6d365776f13ab\": container with ID starting with 2273a440ff8b094cfffc936bf77205754ee60a3b4410bfc11ee6d365776f13ab not found: ID does not exist" containerID="2273a440ff8b094cfffc936bf77205754ee60a3b4410bfc11ee6d365776f13ab" Oct 31 00:47:20.264563 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:20.264527 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2273a440ff8b094cfffc936bf77205754ee60a3b4410bfc11ee6d365776f13ab"} err="failed to get container status \"2273a440ff8b094cfffc936bf77205754ee60a3b4410bfc11ee6d365776f13ab\": rpc error: code = NotFound desc = could not find container \"2273a440ff8b094cfffc936bf77205754ee60a3b4410bfc11ee6d365776f13ab\": container with ID starting with 2273a440ff8b094cfffc936bf77205754ee60a3b4410bfc11ee6d365776f13ab not found: ID does not exist" Oct 31 00:47:20.264563 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:20.264542 5274 scope.go:122] "RemoveContainer" containerID="003e2f64063208af10073e83c74055a6ec87b08bdcf579f15067dc9a92078124" Oct 31 00:47:20.264758 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:47:20.264740 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"003e2f64063208af10073e83c74055a6ec87b08bdcf579f15067dc9a92078124\": container with ID starting with 003e2f64063208af10073e83c74055a6ec87b08bdcf579f15067dc9a92078124 not found: ID does not exist" containerID="003e2f64063208af10073e83c74055a6ec87b08bdcf579f15067dc9a92078124" Oct 31 00:47:20.264812 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:20.264761 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"003e2f64063208af10073e83c74055a6ec87b08bdcf579f15067dc9a92078124"} err="failed to get container status \"003e2f64063208af10073e83c74055a6ec87b08bdcf579f15067dc9a92078124\": rpc error: code = NotFound desc = could not find container \"003e2f64063208af10073e83c74055a6ec87b08bdcf579f15067dc9a92078124\": container with ID starting with 003e2f64063208af10073e83c74055a6ec87b08bdcf579f15067dc9a92078124 not found: ID does not exist" Oct 31 00:47:20.269166 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:20.269130 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.26911808 podStartE2EDuration="2.26911808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:20.267539484 +0000 UTC m=+2876.904358198" watchObservedRunningTime="2026-10-31 00:47:20.26911808 +0000 UTC m=+2876.905936795" Oct 31 00:47:20.286623 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:20.286597 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:47:20.291387 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:20.291365 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:47:20.899333 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:47:20.899294 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:47:21.481366 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:21.481337 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f69d683-62af-468f-8ae5-f2b2d101bf61" path="/var/lib/kubelet/pods/3f69d683-62af-468f-8ae5-f2b2d101bf61/volumes" Oct 31 00:47:24.793317 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:24.793288 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:47:24.793712 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:24.793611 5274 cpu_manager.go:414] "containerMap: removing container" podUID="3f69d683-62af-468f-8ae5-f2b2d101bf61" containerName="config-reloader" Oct 31 00:47:24.793712 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:24.793620 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f69d683-62af-468f-8ae5-f2b2d101bf61" containerName="config-reloader" Oct 31 00:47:24.793712 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:24.793633 5274 cpu_manager.go:414] "containerMap: removing container" podUID="3f69d683-62af-468f-8ae5-f2b2d101bf61" containerName="alertmanager" Oct 31 00:47:24.793712 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:24.793638 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f69d683-62af-468f-8ae5-f2b2d101bf61" containerName="alertmanager" Oct 31 00:47:24.793712 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:24.793647 5274 cpu_manager.go:414] "containerMap: removing container" podUID="3f69d683-62af-468f-8ae5-f2b2d101bf61" containerName="init-config-reloader" Oct 31 00:47:24.793712 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:24.793651 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f69d683-62af-468f-8ae5-f2b2d101bf61" containerName="init-config-reloader" Oct 31 00:47:24.793712 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:24.793704 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f69d683-62af-468f-8ae5-f2b2d101bf61" containerName="alertmanager" Oct 31 00:47:24.793925 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:24.793717 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f69d683-62af-468f-8ae5-f2b2d101bf61" containerName="config-reloader" Oct 31 00:47:24.798476 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:24.798456 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:47:24.801908 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:24.801885 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 31 00:47:24.802085 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:24.801909 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:47:24.813189 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:24.813166 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:47:24.897323 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:24.897301 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 31 00:47:24.897471 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:24.897325 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 31 00:47:24.897471 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:24.897392 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-rkqv7\"" Oct 31 00:47:24.906734 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:24.906713 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:47:24.906814 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:24.906742 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:47:24.906814 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:24.906767 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:47:24.906814 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:24.906803 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:47:24.906915 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:24.906849 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:47:24.906946 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:24.906926 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:47:24.907002 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:24.906944 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kbsn\" (UniqueName: \"kubernetes.io/projected/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-kube-api-access-2kbsn\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:47:25.007911 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.007883 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:47:25.008058 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.007925 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:47:25.008058 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.007949 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:47:25.008058 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.008016 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:47:25.008058 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.008040 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kbsn\" (UniqueName: \"kubernetes.io/projected/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-kube-api-access-2kbsn\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:47:25.008267 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.008099 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:47:25.008267 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.008140 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:47:25.008267 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.008248 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:47:25.010260 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.010232 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:47:25.010428 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.010408 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:47:25.010517 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.010495 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:47:25.010729 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.010715 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:47:25.010883 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.010864 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:47:25.023016 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.022993 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kbsn\" (UniqueName: \"kubernetes.io/projected/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-kube-api-access-2kbsn\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:47:25.115656 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.115637 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:47:25.251488 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.251455 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:47:25.256321 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.256299 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:25.271621 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.271590 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:47:25.310554 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.310527 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2e9aed47-75bf-4ce4-ad8e-21c73da85043-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:25.310655 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.310557 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:25.310655 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.310578 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g7gs\" (UniqueName: \"kubernetes.io/projected/2e9aed47-75bf-4ce4-ad8e-21c73da85043-kube-api-access-9g7gs\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:25.310743 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.310684 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2e9aed47-75bf-4ce4-ad8e-21c73da85043-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:25.310743 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.310726 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2e9aed47-75bf-4ce4-ad8e-21c73da85043-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:25.310819 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.310774 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:25.310861 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.310831 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:25.310899 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.310870 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:25.310899 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.310887 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2e9aed47-75bf-4ce4-ad8e-21c73da85043-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:25.311003 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.310927 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2e9aed47-75bf-4ce4-ad8e-21c73da85043-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:25.311003 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.310983 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:25.311078 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.311050 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:25.336880 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.336784 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 31 00:47:25.336880 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.336810 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 31 00:47:25.336880 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.336822 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 31 00:47:25.336880 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.336851 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 31 00:47:25.336880 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.336863 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 31 00:47:25.337175 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.336911 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:47:25.337175 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.336915 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:47:25.337175 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.336997 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-dlnbv\"" Oct 31 00:47:25.337175 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.337006 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 31 00:47:25.337175 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.337070 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 31 00:47:25.399885 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.399809 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:47:25.402230 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:47:25.402209 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea3bb993_f20b_4a7a_a9dc_f198aafd9a28.slice/crio-03d2a16352077cee0f3956db7012c2c2204381a1730d2d91c57c0b7820fa6b2d WatchSource:0}: Error finding container 03d2a16352077cee0f3956db7012c2c2204381a1730d2d91c57c0b7820fa6b2d: Status 404 returned error can't find the container with id 03d2a16352077cee0f3956db7012c2c2204381a1730d2d91c57c0b7820fa6b2d Oct 31 00:47:25.411707 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.411682 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:25.411776 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.411722 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:25.411776 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.411746 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2e9aed47-75bf-4ce4-ad8e-21c73da85043-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:25.411867 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.411779 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2e9aed47-75bf-4ce4-ad8e-21c73da85043-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:25.411867 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.411818 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:25.411867 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.411844 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:25.412007 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.411879 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2e9aed47-75bf-4ce4-ad8e-21c73da85043-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:25.412007 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.411904 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:25.412007 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.411934 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g7gs\" (UniqueName: \"kubernetes.io/projected/2e9aed47-75bf-4ce4-ad8e-21c73da85043-kube-api-access-9g7gs\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:25.412147 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.412021 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2e9aed47-75bf-4ce4-ad8e-21c73da85043-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:25.412147 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.412067 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2e9aed47-75bf-4ce4-ad8e-21c73da85043-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:25.412147 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.412093 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:25.412493 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.412468 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2e9aed47-75bf-4ce4-ad8e-21c73da85043-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:25.412730 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.412705 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2e9aed47-75bf-4ce4-ad8e-21c73da85043-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:25.414532 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.414503 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:25.414822 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.414800 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:25.414985 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.414937 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:25.415089 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.415039 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:25.415089 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.415069 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2e9aed47-75bf-4ce4-ad8e-21c73da85043-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:25.415250 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.415218 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2e9aed47-75bf-4ce4-ad8e-21c73da85043-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:25.415377 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.415360 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2e9aed47-75bf-4ce4-ad8e-21c73da85043-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:25.415561 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.415543 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:25.416858 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.416839 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:25.424487 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:25.424468 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g7gs\" (UniqueName: \"kubernetes.io/projected/2e9aed47-75bf-4ce4-ad8e-21c73da85043-kube-api-access-9g7gs\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:26.281591 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:26.281557 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28","Type":"ContainerStarted","Data":"a4e42cbcf4e243ecb4bbc1f7b56868a7480cb6716487d7b38b40809cca6baeef"} Oct 31 00:47:26.281976 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:26.281640 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28","Type":"ContainerStarted","Data":"03d2a16352077cee0f3956db7012c2c2204381a1730d2d91c57c0b7820fa6b2d"} Oct 31 00:47:29.297623 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:29.297592 5274 generic.go:356] "Generic (PLEG): container finished" podID="ea3bb993-f20b-4a7a-a9dc-f198aafd9a28" containerID="a4e42cbcf4e243ecb4bbc1f7b56868a7480cb6716487d7b38b40809cca6baeef" exitCode=0 Oct 31 00:47:29.298183 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:29.297671 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28","Type":"ContainerDied","Data":"a4e42cbcf4e243ecb4bbc1f7b56868a7480cb6716487d7b38b40809cca6baeef"} Oct 31 00:47:30.244133 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:30.244102 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lffkt" Oct 31 00:47:30.304619 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:30.304591 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28","Type":"ContainerStarted","Data":"c98d27b2901aaa9d0b4140c9b059347cb35a529c3f569b9d4251d6bc6679ff26"} Oct 31 00:47:30.305095 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:30.304656 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28","Type":"ContainerStarted","Data":"d8edd303852fca685538a22d0e5591290ab6c908e9637d9f9a8994a9ac9a137a"} Oct 31 00:47:30.305095 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:30.304854 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:47:30.334910 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:30.334860 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=3.148741503 podStartE2EDuration="6.334846325s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.186104822s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:30.334686642 +0000 UTC m=+2886.971505369" watchObservedRunningTime="2026-10-31 00:47:30.334846325 +0000 UTC m=+2886.971665040" Oct 31 00:47:31.363488 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:31.363446 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:31.371256 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:31.371218 5274 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:31.463937 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:31.463912 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:31.464057 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:31.464040 5274 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:31.467588 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:31.467572 5274 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") DevicePath \"csi-3beb91b3ebf177571ca474589da3560cd18787cfa50b2a0b9bfef85661e6cf21\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:31.558396 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:31.558367 5274 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/534eb232558c8505fa55558aa87cc96e2d2b0adb946edb717939ec387be54e97/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:31.576477 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:31.576453 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:31.877653 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:31.877626 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:32.161743 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:32.161712 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:47:32.163543 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:47:32.163516 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e9aed47_75bf_4ce4_ad8e_21c73da85043.slice/crio-ba334ebea2802d0134050cc15a6bc58fb127d2841cc27abe1886d16aab982ce6 WatchSource:0}: Error finding container ba334ebea2802d0134050cc15a6bc58fb127d2841cc27abe1886d16aab982ce6: Status 404 returned error can't find the container with id ba334ebea2802d0134050cc15a6bc58fb127d2841cc27abe1886d16aab982ce6 Oct 31 00:47:32.316072 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:32.316033 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"2e9aed47-75bf-4ce4-ad8e-21c73da85043","Type":"ContainerStarted","Data":"ba334ebea2802d0134050cc15a6bc58fb127d2841cc27abe1886d16aab982ce6"} Oct 31 00:47:33.323681 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:33.323647 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"2e9aed47-75bf-4ce4-ad8e-21c73da85043","Type":"ContainerStarted","Data":"3e73958ad1f0f3dfb20668930575bc70cc034d6edc95fb6b63d76438d8d1a640"} Oct 31 00:47:36.310357 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:36.310329 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:47:36.349258 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:36.349222 5274 generic.go:356] "Generic (PLEG): container finished" podID="2e9aed47-75bf-4ce4-ad8e-21c73da85043" containerID="3e73958ad1f0f3dfb20668930575bc70cc034d6edc95fb6b63d76438d8d1a640" exitCode=0 Oct 31 00:47:36.349444 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:36.349343 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"2e9aed47-75bf-4ce4-ad8e-21c73da85043","Type":"ContainerDied","Data":"3e73958ad1f0f3dfb20668930575bc70cc034d6edc95fb6b63d76438d8d1a640"} Oct 31 00:47:37.356737 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:37.356701 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"2e9aed47-75bf-4ce4-ad8e-21c73da85043","Type":"ContainerStarted","Data":"f35f54813c05fc36e01db5a5f10e97a70f7ce37a8899f008a53c3767004040e8"} Oct 31 00:47:37.357151 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:37.356766 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"2e9aed47-75bf-4ce4-ad8e-21c73da85043","Type":"ContainerStarted","Data":"ae640f32786003e41921cfa4d825f2c0aa1ce2fcfcdce402db8608cc541b927f"} Oct 31 00:47:37.357151 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:37.356790 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"2e9aed47-75bf-4ce4-ad8e-21c73da85043","Type":"ContainerStarted","Data":"f574ca9f8f59b1037d24e660a67fd9c9b47f492de20b7de9bb0186c4ad87ecf6"} Oct 31 00:47:37.393381 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:37.393328 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=9.207059225 podStartE2EDuration="12.393316743s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.186257518s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:47:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:37.390859537 +0000 UTC m=+2894.027678252" watchObservedRunningTime="2026-10-31 00:47:37.393316743 +0000 UTC m=+2894.030135509" Oct 31 00:47:41.878604 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:41.878571 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:46.878799 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:46.878764 5274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:46.884018 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:46.883997 5274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:47.417653 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:47.417600 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:47:49.089885 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.089855 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:47:49.090348 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.090120 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="fb305977-92f9-45c8-9f97-e37577330b72" containerName="otc-container" containerID="cri-o://7a645a5352ceb300e6c997ae1b35e5a347a511c7723d43d4a254964e7e30cece" gracePeriod=30 Oct 31 00:47:49.332329 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.332305 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:49.421351 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.421322 5274 generic.go:356] "Generic (PLEG): container finished" podID="fb305977-92f9-45c8-9f97-e37577330b72" containerID="7a645a5352ceb300e6c997ae1b35e5a347a511c7723d43d4a254964e7e30cece" exitCode=0 Oct 31 00:47:49.421542 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.421376 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:49.421542 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.421414 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"fb305977-92f9-45c8-9f97-e37577330b72","Type":"ContainerDied","Data":"7a645a5352ceb300e6c997ae1b35e5a347a511c7723d43d4a254964e7e30cece"} Oct 31 00:47:49.421542 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.421465 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"fb305977-92f9-45c8-9f97-e37577330b72","Type":"ContainerDied","Data":"998c7a078574f8fe578ad15b03525e10c25cca661265d3cc4abf72b805b0f3ad"} Oct 31 00:47:49.421542 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.421488 5274 scope.go:122] "RemoveContainer" containerID="7a645a5352ceb300e6c997ae1b35e5a347a511c7723d43d4a254964e7e30cece" Oct 31 00:47:49.430223 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.430206 5274 scope.go:122] "RemoveContainer" containerID="7a645a5352ceb300e6c997ae1b35e5a347a511c7723d43d4a254964e7e30cece" Oct 31 00:47:49.430486 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:47:49.430467 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a645a5352ceb300e6c997ae1b35e5a347a511c7723d43d4a254964e7e30cece\": container with ID starting with 7a645a5352ceb300e6c997ae1b35e5a347a511c7723d43d4a254964e7e30cece not found: ID does not exist" containerID="7a645a5352ceb300e6c997ae1b35e5a347a511c7723d43d4a254964e7e30cece" Oct 31 00:47:49.430551 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.430498 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a645a5352ceb300e6c997ae1b35e5a347a511c7723d43d4a254964e7e30cece"} err="failed to get container status \"7a645a5352ceb300e6c997ae1b35e5a347a511c7723d43d4a254964e7e30cece\": rpc error: code = NotFound desc = could not find container \"7a645a5352ceb300e6c997ae1b35e5a347a511c7723d43d4a254964e7e30cece\": container with ID starting with 7a645a5352ceb300e6c997ae1b35e5a347a511c7723d43d4a254964e7e30cece not found: ID does not exist" Oct 31 00:47:49.507876 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.507853 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb305977-92f9-45c8-9f97-e37577330b72-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/fb305977-92f9-45c8-9f97-e37577330b72-data-science-collector-ta-client-cert\") pod \"fb305977-92f9-45c8-9f97-e37577330b72\" (UID: \"fb305977-92f9-45c8-9f97-e37577330b72\") " Oct 31 00:47:49.508012 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.507884 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb305977-92f9-45c8-9f97-e37577330b72-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb305977-92f9-45c8-9f97-e37577330b72-otlp-tls-certs\") pod \"fb305977-92f9-45c8-9f97-e37577330b72\" (UID: \"fb305977-92f9-45c8-9f97-e37577330b72\") " Oct 31 00:47:49.508012 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.507928 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb305977-92f9-45c8-9f97-e37577330b72-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb305977-92f9-45c8-9f97-e37577330b72-tls-certs\") pod \"fb305977-92f9-45c8-9f97-e37577330b72\" (UID: \"fb305977-92f9-45c8-9f97-e37577330b72\") " Oct 31 00:47:49.508091 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.508029 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb305977-92f9-45c8-9f97-e37577330b72-kube-api-access-djtsq\" (UniqueName: \"kubernetes.io/projected/fb305977-92f9-45c8-9f97-e37577330b72-kube-api-access-djtsq\") pod \"fb305977-92f9-45c8-9f97-e37577330b72\" (UID: \"fb305977-92f9-45c8-9f97-e37577330b72\") " Oct 31 00:47:49.508091 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.508067 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb305977-92f9-45c8-9f97-e37577330b72-otc-internal\" (UniqueName: \"kubernetes.io/configmap/fb305977-92f9-45c8-9f97-e37577330b72-otc-internal\") pod \"fb305977-92f9-45c8-9f97-e37577330b72\" (UID: \"fb305977-92f9-45c8-9f97-e37577330b72\") " Oct 31 00:47:49.508201 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.508109 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb305977-92f9-45c8-9f97-e37577330b72-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb305977-92f9-45c8-9f97-e37577330b72-monitoring-tls-certs\") pod \"fb305977-92f9-45c8-9f97-e37577330b72\" (UID: \"fb305977-92f9-45c8-9f97-e37577330b72\") " Oct 31 00:47:49.508445 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.508413 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb305977-92f9-45c8-9f97-e37577330b72-otc-internal" pod "fb305977-92f9-45c8-9f97-e37577330b72" (UID: "fb305977-92f9-45c8-9f97-e37577330b72"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:49.510120 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.510095 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb305977-92f9-45c8-9f97-e37577330b72-tls-certs" pod "fb305977-92f9-45c8-9f97-e37577330b72" (UID: "fb305977-92f9-45c8-9f97-e37577330b72"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:49.510120 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.510110 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb305977-92f9-45c8-9f97-e37577330b72-otlp-tls-certs" pod "fb305977-92f9-45c8-9f97-e37577330b72" (UID: "fb305977-92f9-45c8-9f97-e37577330b72"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:49.510245 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.510171 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb305977-92f9-45c8-9f97-e37577330b72-kube-api-access-djtsq" pod "fb305977-92f9-45c8-9f97-e37577330b72" (UID: "fb305977-92f9-45c8-9f97-e37577330b72"). InnerVolumeSpecName "kube-api-access-djtsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:49.510394 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.510375 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb305977-92f9-45c8-9f97-e37577330b72-monitoring-tls-certs" pod "fb305977-92f9-45c8-9f97-e37577330b72" (UID: "fb305977-92f9-45c8-9f97-e37577330b72"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:49.510456 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.510393 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb305977-92f9-45c8-9f97-e37577330b72-data-science-collector-ta-client-cert" pod "fb305977-92f9-45c8-9f97-e37577330b72" (UID: "fb305977-92f9-45c8-9f97-e37577330b72"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:49.608665 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.608644 5274 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/fb305977-92f9-45c8-9f97-e37577330b72-data-science-collector-ta-client-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:49.608665 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.608663 5274 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb305977-92f9-45c8-9f97-e37577330b72-otlp-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:49.608803 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.608673 5274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb305977-92f9-45c8-9f97-e37577330b72-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:49.608803 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.608683 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djtsq\" (UniqueName: \"kubernetes.io/projected/fb305977-92f9-45c8-9f97-e37577330b72-kube-api-access-djtsq\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:49.608803 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.608692 5274 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fb305977-92f9-45c8-9f97-e37577330b72-otc-internal\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:49.608803 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.608700 5274 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb305977-92f9-45c8-9f97-e37577330b72-monitoring-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:49.745140 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.745116 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:47:49.749514 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.749492 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:47:49.777357 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.777332 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:47:49.777674 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.777662 5274 cpu_manager.go:414] "containerMap: removing container" podUID="fb305977-92f9-45c8-9f97-e37577330b72" containerName="otc-container" Oct 31 00:47:49.777716 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.777675 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb305977-92f9-45c8-9f97-e37577330b72" containerName="otc-container" Oct 31 00:47:49.777757 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.777748 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb305977-92f9-45c8-9f97-e37577330b72" containerName="otc-container" Oct 31 00:47:49.782266 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.782250 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:49.786783 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.786764 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:47:49.786866 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.786815 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:47:49.786916 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.786901 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:47:49.787348 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.787333 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:47:49.787390 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.787334 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:47:49.787390 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.787383 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-9ht4b\"" Oct 31 00:47:49.792592 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.792528 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:47:49.911543 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.911513 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3c09a8-2070-40e3-8904-81df69f7848a-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1e3c09a8-2070-40e3-8904-81df69f7848a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:49.911679 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.911553 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1e3c09a8-2070-40e3-8904-81df69f7848a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"1e3c09a8-2070-40e3-8904-81df69f7848a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:49.911679 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.911569 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1e3c09a8-2070-40e3-8904-81df69f7848a-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"1e3c09a8-2070-40e3-8904-81df69f7848a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:49.911774 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.911688 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3c09a8-2070-40e3-8904-81df69f7848a-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1e3c09a8-2070-40e3-8904-81df69f7848a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:49.911774 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.911725 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3c09a8-2070-40e3-8904-81df69f7848a-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1e3c09a8-2070-40e3-8904-81df69f7848a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:49.911774 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:49.911744 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntrh8\" (UniqueName: \"kubernetes.io/projected/1e3c09a8-2070-40e3-8904-81df69f7848a-kube-api-access-ntrh8\") pod \"data-science-collector-collector-0\" (UID: \"1e3c09a8-2070-40e3-8904-81df69f7848a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:50.012229 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:50.012160 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3c09a8-2070-40e3-8904-81df69f7848a-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1e3c09a8-2070-40e3-8904-81df69f7848a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:50.012229 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:50.012203 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1e3c09a8-2070-40e3-8904-81df69f7848a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"1e3c09a8-2070-40e3-8904-81df69f7848a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:50.012405 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:50.012314 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1e3c09a8-2070-40e3-8904-81df69f7848a-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"1e3c09a8-2070-40e3-8904-81df69f7848a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:50.012405 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:50.012352 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3c09a8-2070-40e3-8904-81df69f7848a-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1e3c09a8-2070-40e3-8904-81df69f7848a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:50.012405 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:50.012378 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3c09a8-2070-40e3-8904-81df69f7848a-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1e3c09a8-2070-40e3-8904-81df69f7848a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:50.012567 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:50.012498 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntrh8\" (UniqueName: \"kubernetes.io/projected/1e3c09a8-2070-40e3-8904-81df69f7848a-kube-api-access-ntrh8\") pod \"data-science-collector-collector-0\" (UID: \"1e3c09a8-2070-40e3-8904-81df69f7848a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:50.013070 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:50.013045 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1e3c09a8-2070-40e3-8904-81df69f7848a-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"1e3c09a8-2070-40e3-8904-81df69f7848a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:50.014804 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:50.014786 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3c09a8-2070-40e3-8904-81df69f7848a-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1e3c09a8-2070-40e3-8904-81df69f7848a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:50.014900 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:50.014806 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3c09a8-2070-40e3-8904-81df69f7848a-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1e3c09a8-2070-40e3-8904-81df69f7848a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:50.015006 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:50.014986 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3c09a8-2070-40e3-8904-81df69f7848a-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1e3c09a8-2070-40e3-8904-81df69f7848a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:50.015052 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:50.015025 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1e3c09a8-2070-40e3-8904-81df69f7848a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"1e3c09a8-2070-40e3-8904-81df69f7848a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:50.025384 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:50.025365 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntrh8\" (UniqueName: \"kubernetes.io/projected/1e3c09a8-2070-40e3-8904-81df69f7848a-kube-api-access-ntrh8\") pod \"data-science-collector-collector-0\" (UID: \"1e3c09a8-2070-40e3-8904-81df69f7848a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:50.095999 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:50.095977 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:47:50.375725 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:50.375704 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:47:50.377564 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:47:50.377538 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e3c09a8_2070_40e3_8904_81df69f7848a.slice/crio-57e66b2bb94fa64818d69ee47c65127bdc4c159ae5906babd3a8cf4c9c51eae4 WatchSource:0}: Error finding container 57e66b2bb94fa64818d69ee47c65127bdc4c159ae5906babd3a8cf4c9c51eae4: Status 404 returned error can't find the container with id 57e66b2bb94fa64818d69ee47c65127bdc4c159ae5906babd3a8cf4c9c51eae4 Oct 31 00:47:50.380209 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:50.380175 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:47:50.380319 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:50.380234 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:47:50.428073 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:50.428050 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"1e3c09a8-2070-40e3-8904-81df69f7848a","Type":"ContainerStarted","Data":"57e66b2bb94fa64818d69ee47c65127bdc4c159ae5906babd3a8cf4c9c51eae4"} Oct 31 00:47:50.899150 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:47:50.899117 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:47:51.434925 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:51.434885 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"1e3c09a8-2070-40e3-8904-81df69f7848a","Type":"ContainerStarted","Data":"50c12b5c8e0635dc57976413fcddb76eb5ba8ae1c0346a657b5859ee3b4c3f52"} Oct 31 00:47:51.457066 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:51.457018 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.457007248 podStartE2EDuration="2.457007248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:51.456405484 +0000 UTC m=+2908.093224201" watchObservedRunningTime="2026-10-31 00:47:51.457007248 +0000 UTC m=+2908.093825960" Oct 31 00:47:51.482121 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:47:51.482092 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb305977-92f9-45c8-9f97-e37577330b72" path="/var/lib/kubelet/pods/fb305977-92f9-45c8-9f97-e37577330b72/volumes" Oct 31 00:48:07.178801 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:07.178766 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:48:07.179348 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:07.178992 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="1e3c09a8-2070-40e3-8904-81df69f7848a" containerName="otc-container" containerID="cri-o://50c12b5c8e0635dc57976413fcddb76eb5ba8ae1c0346a657b5859ee3b4c3f52" gracePeriod=30 Oct 31 00:48:09.520998 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.520973 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:09.522801 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.522776 5274 generic.go:356] "Generic (PLEG): container finished" podID="1e3c09a8-2070-40e3-8904-81df69f7848a" containerID="50c12b5c8e0635dc57976413fcddb76eb5ba8ae1c0346a657b5859ee3b4c3f52" exitCode=0 Oct 31 00:48:09.522926 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.522813 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:09.522926 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.522855 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"1e3c09a8-2070-40e3-8904-81df69f7848a","Type":"ContainerDied","Data":"50c12b5c8e0635dc57976413fcddb76eb5ba8ae1c0346a657b5859ee3b4c3f52"} Oct 31 00:48:09.522926 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.522905 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"1e3c09a8-2070-40e3-8904-81df69f7848a","Type":"ContainerDied","Data":"57e66b2bb94fa64818d69ee47c65127bdc4c159ae5906babd3a8cf4c9c51eae4"} Oct 31 00:48:09.523068 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.522935 5274 scope.go:122] "RemoveContainer" containerID="50c12b5c8e0635dc57976413fcddb76eb5ba8ae1c0346a657b5859ee3b4c3f52" Oct 31 00:48:09.531116 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.531099 5274 scope.go:122] "RemoveContainer" containerID="50c12b5c8e0635dc57976413fcddb76eb5ba8ae1c0346a657b5859ee3b4c3f52" Oct 31 00:48:09.531387 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:48:09.531369 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50c12b5c8e0635dc57976413fcddb76eb5ba8ae1c0346a657b5859ee3b4c3f52\": container with ID starting with 50c12b5c8e0635dc57976413fcddb76eb5ba8ae1c0346a657b5859ee3b4c3f52 not found: ID does not exist" containerID="50c12b5c8e0635dc57976413fcddb76eb5ba8ae1c0346a657b5859ee3b4c3f52" Oct 31 00:48:09.531446 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.531398 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50c12b5c8e0635dc57976413fcddb76eb5ba8ae1c0346a657b5859ee3b4c3f52"} err="failed to get container status \"50c12b5c8e0635dc57976413fcddb76eb5ba8ae1c0346a657b5859ee3b4c3f52\": rpc error: code = NotFound desc = could not find container \"50c12b5c8e0635dc57976413fcddb76eb5ba8ae1c0346a657b5859ee3b4c3f52\": container with ID starting with 50c12b5c8e0635dc57976413fcddb76eb5ba8ae1c0346a657b5859ee3b4c3f52 not found: ID does not exist" Oct 31 00:48:09.556948 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.556884 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e3c09a8-2070-40e3-8904-81df69f7848a-otc-internal\" (UniqueName: \"kubernetes.io/configmap/1e3c09a8-2070-40e3-8904-81df69f7848a-otc-internal\") pod \"1e3c09a8-2070-40e3-8904-81df69f7848a\" (UID: \"1e3c09a8-2070-40e3-8904-81df69f7848a\") " Oct 31 00:48:09.556948 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.556926 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e3c09a8-2070-40e3-8904-81df69f7848a-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3c09a8-2070-40e3-8904-81df69f7848a-tls-certs\") pod \"1e3c09a8-2070-40e3-8904-81df69f7848a\" (UID: \"1e3c09a8-2070-40e3-8904-81df69f7848a\") " Oct 31 00:48:09.557145 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.556978 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e3c09a8-2070-40e3-8904-81df69f7848a-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1e3c09a8-2070-40e3-8904-81df69f7848a-data-science-collector-ta-client-cert\") pod \"1e3c09a8-2070-40e3-8904-81df69f7848a\" (UID: \"1e3c09a8-2070-40e3-8904-81df69f7848a\") " Oct 31 00:48:09.557145 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.557032 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e3c09a8-2070-40e3-8904-81df69f7848a-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3c09a8-2070-40e3-8904-81df69f7848a-monitoring-tls-certs\") pod \"1e3c09a8-2070-40e3-8904-81df69f7848a\" (UID: \"1e3c09a8-2070-40e3-8904-81df69f7848a\") " Oct 31 00:48:09.557145 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.557074 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e3c09a8-2070-40e3-8904-81df69f7848a-kube-api-access-ntrh8\" (UniqueName: \"kubernetes.io/projected/1e3c09a8-2070-40e3-8904-81df69f7848a-kube-api-access-ntrh8\") pod \"1e3c09a8-2070-40e3-8904-81df69f7848a\" (UID: \"1e3c09a8-2070-40e3-8904-81df69f7848a\") " Oct 31 00:48:09.557145 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.557122 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e3c09a8-2070-40e3-8904-81df69f7848a-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3c09a8-2070-40e3-8904-81df69f7848a-otlp-tls-certs\") pod \"1e3c09a8-2070-40e3-8904-81df69f7848a\" (UID: \"1e3c09a8-2070-40e3-8904-81df69f7848a\") " Oct 31 00:48:09.557734 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.557709 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e3c09a8-2070-40e3-8904-81df69f7848a-otc-internal" pod "1e3c09a8-2070-40e3-8904-81df69f7848a" (UID: "1e3c09a8-2070-40e3-8904-81df69f7848a"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:09.559472 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.559430 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e3c09a8-2070-40e3-8904-81df69f7848a-data-science-collector-ta-client-cert" pod "1e3c09a8-2070-40e3-8904-81df69f7848a" (UID: "1e3c09a8-2070-40e3-8904-81df69f7848a"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:09.559630 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.559609 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e3c09a8-2070-40e3-8904-81df69f7848a-monitoring-tls-certs" pod "1e3c09a8-2070-40e3-8904-81df69f7848a" (UID: "1e3c09a8-2070-40e3-8904-81df69f7848a"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:09.559777 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.559753 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e3c09a8-2070-40e3-8904-81df69f7848a-kube-api-access-ntrh8" pod "1e3c09a8-2070-40e3-8904-81df69f7848a" (UID: "1e3c09a8-2070-40e3-8904-81df69f7848a"). InnerVolumeSpecName "kube-api-access-ntrh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:09.559891 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.559870 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e3c09a8-2070-40e3-8904-81df69f7848a-otlp-tls-certs" pod "1e3c09a8-2070-40e3-8904-81df69f7848a" (UID: "1e3c09a8-2070-40e3-8904-81df69f7848a"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:09.560167 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.560148 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e3c09a8-2070-40e3-8904-81df69f7848a-tls-certs" pod "1e3c09a8-2070-40e3-8904-81df69f7848a" (UID: "1e3c09a8-2070-40e3-8904-81df69f7848a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:09.658267 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.658244 5274 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3c09a8-2070-40e3-8904-81df69f7848a-monitoring-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:09.658267 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.658266 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntrh8\" (UniqueName: \"kubernetes.io/projected/1e3c09a8-2070-40e3-8904-81df69f7848a-kube-api-access-ntrh8\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:09.658401 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.658276 5274 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3c09a8-2070-40e3-8904-81df69f7848a-otlp-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:09.658401 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.658285 5274 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1e3c09a8-2070-40e3-8904-81df69f7848a-otc-internal\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:09.658401 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.658293 5274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3c09a8-2070-40e3-8904-81df69f7848a-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:09.658401 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.658301 5274 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1e3c09a8-2070-40e3-8904-81df69f7848a-data-science-collector-ta-client-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:09.845595 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.845563 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:48:09.849212 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.849180 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:48:09.877799 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.877780 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:48:09.878163 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.878150 5274 cpu_manager.go:414] "containerMap: removing container" podUID="1e3c09a8-2070-40e3-8904-81df69f7848a" containerName="otc-container" Oct 31 00:48:09.878216 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.878164 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e3c09a8-2070-40e3-8904-81df69f7848a" containerName="otc-container" Oct 31 00:48:09.878466 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.878241 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e3c09a8-2070-40e3-8904-81df69f7848a" containerName="otc-container" Oct 31 00:48:09.882825 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.882805 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:09.887526 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.887504 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:48:09.887526 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.887521 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:48:09.887652 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.887547 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-9ht4b\"" Oct 31 00:48:09.887652 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.887617 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:48:09.887719 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.887662 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:48:09.887752 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.887723 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:48:09.893781 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.893759 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:48:09.960677 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.960655 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4g8t\" (UniqueName: \"kubernetes.io/projected/b58466ba-9445-4412-b86b-05110f21502f-kube-api-access-d4g8t\") pod \"data-science-collector-collector-0\" (UID: \"b58466ba-9445-4412-b86b-05110f21502f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:09.960816 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.960688 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"b58466ba-9445-4412-b86b-05110f21502f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:09.960816 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.960733 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b58466ba-9445-4412-b86b-05110f21502f-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"b58466ba-9445-4412-b86b-05110f21502f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:09.960895 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.960847 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"b58466ba-9445-4412-b86b-05110f21502f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:09.960895 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.960868 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"b58466ba-9445-4412-b86b-05110f21502f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:09.960980 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:09.960894 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"b58466ba-9445-4412-b86b-05110f21502f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:10.061661 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.061633 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"b58466ba-9445-4412-b86b-05110f21502f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:10.061796 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.061694 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4g8t\" (UniqueName: \"kubernetes.io/projected/b58466ba-9445-4412-b86b-05110f21502f-kube-api-access-d4g8t\") pod \"data-science-collector-collector-0\" (UID: \"b58466ba-9445-4412-b86b-05110f21502f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:10.061796 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.061736 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"b58466ba-9445-4412-b86b-05110f21502f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:10.061882 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.061813 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b58466ba-9445-4412-b86b-05110f21502f-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"b58466ba-9445-4412-b86b-05110f21502f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:10.061935 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.061911 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"b58466ba-9445-4412-b86b-05110f21502f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:10.062040 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.061972 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"b58466ba-9445-4412-b86b-05110f21502f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:10.062592 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.062567 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b58466ba-9445-4412-b86b-05110f21502f-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"b58466ba-9445-4412-b86b-05110f21502f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:10.064249 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.064228 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"b58466ba-9445-4412-b86b-05110f21502f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:10.064395 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.064376 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"b58466ba-9445-4412-b86b-05110f21502f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:10.064395 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.064389 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"b58466ba-9445-4412-b86b-05110f21502f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:10.064520 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.064471 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"b58466ba-9445-4412-b86b-05110f21502f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:10.074173 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.074149 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4g8t\" (UniqueName: \"kubernetes.io/projected/b58466ba-9445-4412-b86b-05110f21502f-kube-api-access-d4g8t\") pod \"data-science-collector-collector-0\" (UID: \"b58466ba-9445-4412-b86b-05110f21502f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:10.196155 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.196128 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:10.306625 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.306582 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-lffkt"] Oct 31 00:48:10.306852 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.306817 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lffkt" podUID="46200120-336e-49c2-81f2-fc7a540ca5df" containerName="korrel8r" containerID="cri-o://c081728fea9b28106665289c08489a5b4fea86ec128f6f918ba0a133fcac70ca" gracePeriod=30 Oct 31 00:48:10.491701 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.489003 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:48:10.505114 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:48:10.505071 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb58466ba_9445_4412_b86b_05110f21502f.slice/crio-5ec6c24b25ea071f13624420dc77eeb2d6bb9acf2f4ff0c6a15874d8ae82b227 WatchSource:0}: Error finding container 5ec6c24b25ea071f13624420dc77eeb2d6bb9acf2f4ff0c6a15874d8ae82b227: Status 404 returned error can't find the container with id 5ec6c24b25ea071f13624420dc77eeb2d6bb9acf2f4ff0c6a15874d8ae82b227 Oct 31 00:48:10.506988 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.506922 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:48:10.507083 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.507035 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:48:10.522408 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.522382 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:48:10.522847 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.522671 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="ea3bb993-f20b-4a7a-a9dc-f198aafd9a28" containerName="alertmanager" containerID="cri-o://d8edd303852fca685538a22d0e5591290ab6c908e9637d9f9a8994a9ac9a137a" gracePeriod=120 Oct 31 00:48:10.523112 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.523071 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="ea3bb993-f20b-4a7a-a9dc-f198aafd9a28" containerName="config-reloader" containerID="cri-o://c98d27b2901aaa9d0b4140c9b059347cb35a529c3f569b9d4251d6bc6679ff26" gracePeriod=120 Oct 31 00:48:10.534347 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.534317 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:48:10.534644 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.534616 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="2e9aed47-75bf-4ce4-ad8e-21c73da85043" containerName="prometheus" containerID="cri-o://f574ca9f8f59b1037d24e660a67fd9c9b47f492de20b7de9bb0186c4ad87ecf6" gracePeriod=600 Oct 31 00:48:10.534827 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.534800 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="2e9aed47-75bf-4ce4-ad8e-21c73da85043" containerName="thanos-sidecar" containerID="cri-o://f35f54813c05fc36e01db5a5f10e97a70f7ce37a8899f008a53c3767004040e8" gracePeriod=600 Oct 31 00:48:10.534923 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.534902 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="2e9aed47-75bf-4ce4-ad8e-21c73da85043" containerName="config-reloader" containerID="cri-o://ae640f32786003e41921cfa4d825f2c0aa1ce2fcfcdce402db8608cc541b927f" gracePeriod=600 Oct 31 00:48:10.544083 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.543983 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"b58466ba-9445-4412-b86b-05110f21502f","Type":"ContainerStarted","Data":"5ec6c24b25ea071f13624420dc77eeb2d6bb9acf2f4ff0c6a15874d8ae82b227"} Oct 31 00:48:10.546646 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.546126 5274 generic.go:356] "Generic (PLEG): container finished" podID="46200120-336e-49c2-81f2-fc7a540ca5df" containerID="c081728fea9b28106665289c08489a5b4fea86ec128f6f918ba0a133fcac70ca" exitCode=0 Oct 31 00:48:10.546646 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.546202 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lffkt" event={"ID":"46200120-336e-49c2-81f2-fc7a540ca5df","Type":"ContainerDied","Data":"c081728fea9b28106665289c08489a5b4fea86ec128f6f918ba0a133fcac70ca"} Oct 31 00:48:10.719169 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.719135 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:48:10.722774 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.722750 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lffkt" Oct 31 00:48:10.770437 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.770352 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46200120-336e-49c2-81f2-fc7a540ca5df-tls\" (UniqueName: \"kubernetes.io/secret/46200120-336e-49c2-81f2-fc7a540ca5df-tls\") pod \"46200120-336e-49c2-81f2-fc7a540ca5df\" (UID: \"46200120-336e-49c2-81f2-fc7a540ca5df\") " Oct 31 00:48:10.770437 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.770417 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46200120-336e-49c2-81f2-fc7a540ca5df-kube-api-access-pn64v\" (UniqueName: \"kubernetes.io/projected/46200120-336e-49c2-81f2-fc7a540ca5df-kube-api-access-pn64v\") pod \"46200120-336e-49c2-81f2-fc7a540ca5df\" (UID: \"46200120-336e-49c2-81f2-fc7a540ca5df\") " Oct 31 00:48:10.770665 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.770473 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46200120-336e-49c2-81f2-fc7a540ca5df-config\" (UniqueName: \"kubernetes.io/configmap/46200120-336e-49c2-81f2-fc7a540ca5df-config\") pod \"46200120-336e-49c2-81f2-fc7a540ca5df\" (UID: \"46200120-336e-49c2-81f2-fc7a540ca5df\") " Oct 31 00:48:10.771914 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.771886 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46200120-336e-49c2-81f2-fc7a540ca5df-config" pod "46200120-336e-49c2-81f2-fc7a540ca5df" (UID: "46200120-336e-49c2-81f2-fc7a540ca5df"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:10.776496 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.775397 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46200120-336e-49c2-81f2-fc7a540ca5df-kube-api-access-pn64v" pod "46200120-336e-49c2-81f2-fc7a540ca5df" (UID: "46200120-336e-49c2-81f2-fc7a540ca5df"). InnerVolumeSpecName "kube-api-access-pn64v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:10.776719 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.776687 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46200120-336e-49c2-81f2-fc7a540ca5df-tls" pod "46200120-336e-49c2-81f2-fc7a540ca5df" (UID: "46200120-336e-49c2-81f2-fc7a540ca5df"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:10.868189 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.868153 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:10.871621 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.871595 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/46200120-336e-49c2-81f2-fc7a540ca5df-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:10.871621 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.871621 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn64v\" (UniqueName: \"kubernetes.io/projected/46200120-336e-49c2-81f2-fc7a540ca5df-kube-api-access-pn64v\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:10.871851 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.871630 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/46200120-336e-49c2-81f2-fc7a540ca5df-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:10.972560 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.972526 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e9aed47-75bf-4ce4-ad8e-21c73da85043-config-out\" (UniqueName: \"kubernetes.io/empty-dir/2e9aed47-75bf-4ce4-ad8e-21c73da85043-config-out\") pod \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " Oct 31 00:48:10.972747 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.972591 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-thanos-prometheus-http-client-file\") pod \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " Oct 31 00:48:10.972747 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.972624 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e9aed47-75bf-4ce4-ad8e-21c73da85043-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2e9aed47-75bf-4ce4-ad8e-21c73da85043-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " Oct 31 00:48:10.972747 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.972705 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " Oct 31 00:48:10.972747 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.972726 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e9aed47-75bf-4ce4-ad8e-21c73da85043-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2e9aed47-75bf-4ce4-ad8e-21c73da85043-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " Oct 31 00:48:10.973092 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.972753 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-config\" (UniqueName: \"kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-config\") pod \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " Oct 31 00:48:10.973092 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.972774 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e9aed47-75bf-4ce4-ad8e-21c73da85043-tls-assets\" (UniqueName: \"kubernetes.io/projected/2e9aed47-75bf-4ce4-ad8e-21c73da85043-tls-assets\") pod \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " Oct 31 00:48:10.973092 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.972798 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-secret-prometheus-web-tls-ca\") pod \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " Oct 31 00:48:10.973092 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.972838 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e9aed47-75bf-4ce4-ad8e-21c73da85043-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2e9aed47-75bf-4ce4-ad8e-21c73da85043-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " Oct 31 00:48:10.973092 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.972873 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e9aed47-75bf-4ce4-ad8e-21c73da85043-kube-api-access-9g7gs\" (UniqueName: \"kubernetes.io/projected/2e9aed47-75bf-4ce4-ad8e-21c73da85043-kube-api-access-9g7gs\") pod \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " Oct 31 00:48:10.973092 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.972910 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-web-config\" (UniqueName: \"kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-web-config\") pod \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " Oct 31 00:48:10.973092 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.972965 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " Oct 31 00:48:10.973092 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.973011 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\" (UID: \"2e9aed47-75bf-4ce4-ad8e-21c73da85043\") " Oct 31 00:48:10.975586 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.975552 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e9aed47-75bf-4ce4-ad8e-21c73da85043-prometheus-data-science-monitoringstack-rulefiles-0" pod "2e9aed47-75bf-4ce4-ad8e-21c73da85043" (UID: "2e9aed47-75bf-4ce4-ad8e-21c73da85043"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:10.975879 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.975857 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e9aed47-75bf-4ce4-ad8e-21c73da85043-prometheus-data-science-monitoringstack-rulefiles-2" pod "2e9aed47-75bf-4ce4-ad8e-21c73da85043" (UID: "2e9aed47-75bf-4ce4-ad8e-21c73da85043"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:10.977006 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.976871 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e9aed47-75bf-4ce4-ad8e-21c73da85043-prometheus-data-science-monitoringstack-rulefiles-1" pod "2e9aed47-75bf-4ce4-ad8e-21c73da85043" (UID: "2e9aed47-75bf-4ce4-ad8e-21c73da85043"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:10.977006 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.976945 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e9aed47-75bf-4ce4-ad8e-21c73da85043-config-out" pod "2e9aed47-75bf-4ce4-ad8e-21c73da85043" (UID: "2e9aed47-75bf-4ce4-ad8e-21c73da85043"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:10.978152 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.978120 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "2e9aed47-75bf-4ce4-ad8e-21c73da85043" (UID: "2e9aed47-75bf-4ce4-ad8e-21c73da85043"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:10.979692 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.979666 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-config" pod "2e9aed47-75bf-4ce4-ad8e-21c73da85043" (UID: "2e9aed47-75bf-4ce4-ad8e-21c73da85043"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:10.981995 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.981942 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e9aed47-75bf-4ce4-ad8e-21c73da85043-tls-assets" pod "2e9aed47-75bf-4ce4-ad8e-21c73da85043" (UID: "2e9aed47-75bf-4ce4-ad8e-21c73da85043"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:10.982376 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.982350 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-secret-prometheus-web-tls-ca" pod "2e9aed47-75bf-4ce4-ad8e-21c73da85043" (UID: "2e9aed47-75bf-4ce4-ad8e-21c73da85043"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:10.983568 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.983539 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-thanos-prometheus-http-client-file" pod "2e9aed47-75bf-4ce4-ad8e-21c73da85043" (UID: "2e9aed47-75bf-4ce4-ad8e-21c73da85043"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:10.983792 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.983768 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e9aed47-75bf-4ce4-ad8e-21c73da85043-kube-api-access-9g7gs" pod "2e9aed47-75bf-4ce4-ad8e-21c73da85043" (UID: "2e9aed47-75bf-4ce4-ad8e-21c73da85043"). InnerVolumeSpecName "kube-api-access-9g7gs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:10.983939 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.983917 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "2e9aed47-75bf-4ce4-ad8e-21c73da85043" (UID: "2e9aed47-75bf-4ce4-ad8e-21c73da85043"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:10.987458 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.987437 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a" pod "2e9aed47-75bf-4ce4-ad8e-21c73da85043" (UID: "2e9aed47-75bf-4ce4-ad8e-21c73da85043"). InnerVolumeSpecName "pvc-63e34709-8d85-4142-b53a-f655698606ff". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:48:10.999021 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:10.998999 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-web-config" pod "2e9aed47-75bf-4ce4-ad8e-21c73da85043" (UID: "2e9aed47-75bf-4ce4-ad8e-21c73da85043"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:11.074523 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.074455 5274 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.074523 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.074484 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2e9aed47-75bf-4ce4-ad8e-21c73da85043-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.074523 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.074513 5274 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" " Oct 31 00:48:11.074523 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.074523 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2e9aed47-75bf-4ce4-ad8e-21c73da85043-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.074738 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.074533 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.074738 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.074544 5274 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2e9aed47-75bf-4ce4-ad8e-21c73da85043-tls-assets\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.074738 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.074552 5274 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-secret-prometheus-web-tls-ca\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.074738 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.074561 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2e9aed47-75bf-4ce4-ad8e-21c73da85043-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.074738 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.074570 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g7gs\" (UniqueName: \"kubernetes.io/projected/2e9aed47-75bf-4ce4-ad8e-21c73da85043-kube-api-access-9g7gs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.074738 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.074581 5274 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-web-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.074738 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.074591 5274 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.074738 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.074601 5274 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/2e9aed47-75bf-4ce4-ad8e-21c73da85043-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.074738 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.074610 5274 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2e9aed47-75bf-4ce4-ad8e-21c73da85043-config-out\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.085563 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.085541 5274 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-63e34709-8d85-4142-b53a-f655698606ff" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a") on node "ip-10-0-129-70.us-east-2.compute.internal" Oct 31 00:48:11.175669 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.175645 5274 reconciler_common.go:299] "Volume detached for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"csi-3beb91b3ebf177571ca474589da3560cd18787cfa50b2a0b9bfef85661e6cf21\"" Oct 31 00:48:11.379881 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.379858 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:48:11.477049 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.477022 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-tls-assets\" (UniqueName: \"kubernetes.io/projected/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-tls-assets\") pod \"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28\" (UID: \"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28\") " Oct 31 00:48:11.477175 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.477062 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-config-volume\" (UniqueName: \"kubernetes.io/secret/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-config-volume\") pod \"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28\" (UID: \"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28\") " Oct 31 00:48:11.477175 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.477099 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-alertmanager-data-science-monitoringstack-db\") pod \"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28\" (UID: \"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28\") " Oct 31 00:48:11.477175 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.477133 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-web-config\" (UniqueName: \"kubernetes.io/secret/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-web-config\") pod \"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28\" (UID: \"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28\") " Oct 31 00:48:11.477347 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.477222 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-cluster-tls-config\") pod \"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28\" (UID: \"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28\") " Oct 31 00:48:11.477347 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.477252 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-config-out\" (UniqueName: \"kubernetes.io/empty-dir/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-config-out\") pod \"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28\" (UID: \"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28\") " Oct 31 00:48:11.477347 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.477308 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-kube-api-access-2kbsn\" (UniqueName: \"kubernetes.io/projected/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-kube-api-access-2kbsn\") pod \"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28\" (UID: \"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28\") " Oct 31 00:48:11.477524 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.477498 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-alertmanager-data-science-monitoringstack-db" pod "ea3bb993-f20b-4a7a-a9dc-f198aafd9a28" (UID: "ea3bb993-f20b-4a7a-a9dc-f198aafd9a28"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:11.477603 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.477589 5274 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.479514 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.479484 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-config-volume" pod "ea3bb993-f20b-4a7a-a9dc-f198aafd9a28" (UID: "ea3bb993-f20b-4a7a-a9dc-f198aafd9a28"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:11.479636 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.479617 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-tls-assets" pod "ea3bb993-f20b-4a7a-a9dc-f198aafd9a28" (UID: "ea3bb993-f20b-4a7a-a9dc-f198aafd9a28"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:11.479855 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.479832 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-kube-api-access-2kbsn" pod "ea3bb993-f20b-4a7a-a9dc-f198aafd9a28" (UID: "ea3bb993-f20b-4a7a-a9dc-f198aafd9a28"). InnerVolumeSpecName "kube-api-access-2kbsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:11.480112 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.480085 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-config-out" pod "ea3bb993-f20b-4a7a-a9dc-f198aafd9a28" (UID: "ea3bb993-f20b-4a7a-a9dc-f198aafd9a28"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:11.482476 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.482451 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e3c09a8-2070-40e3-8904-81df69f7848a" path="/var/lib/kubelet/pods/1e3c09a8-2070-40e3-8904-81df69f7848a/volumes" Oct 31 00:48:11.484771 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.484742 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-cluster-tls-config" pod "ea3bb993-f20b-4a7a-a9dc-f198aafd9a28" (UID: "ea3bb993-f20b-4a7a-a9dc-f198aafd9a28"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:11.492173 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.492154 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-web-config" pod "ea3bb993-f20b-4a7a-a9dc-f198aafd9a28" (UID: "ea3bb993-f20b-4a7a-a9dc-f198aafd9a28"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:11.552484 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.552454 5274 generic.go:356] "Generic (PLEG): container finished" podID="ea3bb993-f20b-4a7a-a9dc-f198aafd9a28" containerID="c98d27b2901aaa9d0b4140c9b059347cb35a529c3f569b9d4251d6bc6679ff26" exitCode=0 Oct 31 00:48:11.552484 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.552473 5274 generic.go:356] "Generic (PLEG): container finished" podID="ea3bb993-f20b-4a7a-a9dc-f198aafd9a28" containerID="d8edd303852fca685538a22d0e5591290ab6c908e9637d9f9a8994a9ac9a137a" exitCode=0 Oct 31 00:48:11.552883 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.552524 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:48:11.552883 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.552538 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28","Type":"ContainerDied","Data":"c98d27b2901aaa9d0b4140c9b059347cb35a529c3f569b9d4251d6bc6679ff26"} Oct 31 00:48:11.552883 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.552592 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28","Type":"ContainerDied","Data":"d8edd303852fca685538a22d0e5591290ab6c908e9637d9f9a8994a9ac9a137a"} Oct 31 00:48:11.552883 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.552614 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ea3bb993-f20b-4a7a-a9dc-f198aafd9a28","Type":"ContainerDied","Data":"03d2a16352077cee0f3956db7012c2c2204381a1730d2d91c57c0b7820fa6b2d"} Oct 31 00:48:11.552883 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.552645 5274 scope.go:122] "RemoveContainer" containerID="c98d27b2901aaa9d0b4140c9b059347cb35a529c3f569b9d4251d6bc6679ff26" Oct 31 00:48:11.554450 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.554424 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"b58466ba-9445-4412-b86b-05110f21502f","Type":"ContainerStarted","Data":"25a69517316026d02a918798b389e3ad66bd25d2216f9c4644e347ab5cdfcb89"} Oct 31 00:48:11.554576 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.554539 5274 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="redhat-ods-monitoring/data-science-collector-collector-0" secret="" err="secret \"data-science-collector-collector-dockercfg-9ht4b\" not found" Oct 31 00:48:11.555875 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.555859 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lffkt" Oct 31 00:48:11.556006 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.555888 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lffkt" event={"ID":"46200120-336e-49c2-81f2-fc7a540ca5df","Type":"ContainerDied","Data":"b88ed78855953b70ef3062bc9a11870c8e137b1c5be64c468cc3bdc12ae30fb6"} Oct 31 00:48:11.558509 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.558420 5274 generic.go:356] "Generic (PLEG): container finished" podID="2e9aed47-75bf-4ce4-ad8e-21c73da85043" containerID="f35f54813c05fc36e01db5a5f10e97a70f7ce37a8899f008a53c3767004040e8" exitCode=0 Oct 31 00:48:11.558509 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.558443 5274 generic.go:356] "Generic (PLEG): container finished" podID="2e9aed47-75bf-4ce4-ad8e-21c73da85043" containerID="ae640f32786003e41921cfa4d825f2c0aa1ce2fcfcdce402db8608cc541b927f" exitCode=0 Oct 31 00:48:11.558509 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.558451 5274 generic.go:356] "Generic (PLEG): container finished" podID="2e9aed47-75bf-4ce4-ad8e-21c73da85043" containerID="f574ca9f8f59b1037d24e660a67fd9c9b47f492de20b7de9bb0186c4ad87ecf6" exitCode=0 Oct 31 00:48:11.558509 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.558477 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"2e9aed47-75bf-4ce4-ad8e-21c73da85043","Type":"ContainerDied","Data":"f35f54813c05fc36e01db5a5f10e97a70f7ce37a8899f008a53c3767004040e8"} Oct 31 00:48:11.558509 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.558496 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:11.558891 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.558542 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"2e9aed47-75bf-4ce4-ad8e-21c73da85043","Type":"ContainerDied","Data":"ae640f32786003e41921cfa4d825f2c0aa1ce2fcfcdce402db8608cc541b927f"} Oct 31 00:48:11.558891 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.558583 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"2e9aed47-75bf-4ce4-ad8e-21c73da85043","Type":"ContainerDied","Data":"f574ca9f8f59b1037d24e660a67fd9c9b47f492de20b7de9bb0186c4ad87ecf6"} Oct 31 00:48:11.558891 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.558621 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"2e9aed47-75bf-4ce4-ad8e-21c73da85043","Type":"ContainerDied","Data":"ba334ebea2802d0134050cc15a6bc58fb127d2841cc27abe1886d16aab982ce6"} Oct 31 00:48:11.563210 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.563191 5274 scope.go:122] "RemoveContainer" containerID="d8edd303852fca685538a22d0e5591290ab6c908e9637d9f9a8994a9ac9a137a" Oct 31 00:48:11.570570 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.570554 5274 scope.go:122] "RemoveContainer" containerID="a4e42cbcf4e243ecb4bbc1f7b56868a7480cb6716487d7b38b40809cca6baeef" Oct 31 00:48:11.578710 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.578689 5274 scope.go:122] "RemoveContainer" containerID="c98d27b2901aaa9d0b4140c9b059347cb35a529c3f569b9d4251d6bc6679ff26" Oct 31 00:48:11.579599 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.579564 5274 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-tls-assets\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.579599 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.579590 5274 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-config-volume\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.579797 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.579604 5274 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-web-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.579797 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.579618 5274 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-cluster-tls-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.579797 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.579632 5274 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-config-out\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.579797 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.579645 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kbsn\" (UniqueName: \"kubernetes.io/projected/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28-kube-api-access-2kbsn\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.579797 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:48:11.579773 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-monitoring-tls: secret "data-science-collector-monitoring-tls" not found Oct 31 00:48:11.580152 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:48:11.579826 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-monitoring-tls-certs podName:b58466ba-9445-4412-b86b-05110f21502f nodeName:}" failed. No retries permitted until 2026-10-31 00:48:12.079807739 +0000 UTC m=+2928.716626445 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-tls-certs" (UniqueName: "kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-monitoring-tls-certs") pod "data-science-collector-collector-0" (UID: "b58466ba-9445-4412-b86b-05110f21502f") : secret "data-science-collector-monitoring-tls" not found Oct 31 00:48:11.580303 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:48:11.580284 5274 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-collector-collector-a0f9182a: configmap "data-science-collector-collector-a0f9182a" not found Oct 31 00:48:11.580358 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:48:11.580341 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b58466ba-9445-4412-b86b-05110f21502f-otc-internal podName:b58466ba-9445-4412-b86b-05110f21502f nodeName:}" failed. No retries permitted until 2026-10-31 00:48:12.080325141 +0000 UTC m=+2928.717143848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otc-internal" (UniqueName: "kubernetes.io/configmap/b58466ba-9445-4412-b86b-05110f21502f-otc-internal") pod "data-science-collector-collector-0" (UID: "b58466ba-9445-4412-b86b-05110f21502f") : configmap "data-science-collector-collector-a0f9182a" not found Oct 31 00:48:11.580413 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:48:11.580379 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-tls: secret "data-science-collector-tls" not found Oct 31 00:48:11.580467 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:48:11.580412 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found Oct 31 00:48:11.580467 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:48:11.580433 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-tls-certs podName:b58466ba-9445-4412-b86b-05110f21502f nodeName:}" failed. No retries permitted until 2026-10-31 00:48:12.080418024 +0000 UTC m=+2928.717236731 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-tls-certs") pod "data-science-collector-collector-0" (UID: "b58466ba-9445-4412-b86b-05110f21502f") : secret "data-science-collector-tls" not found Oct 31 00:48:11.580467 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:48:11.580454 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-otlp-tls-certs podName:b58466ba-9445-4412-b86b-05110f21502f nodeName:}" failed. No retries permitted until 2026-10-31 00:48:12.080442062 +0000 UTC m=+2928.717260771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-otlp-tls-certs") pod "data-science-collector-collector-0" (UID: "b58466ba-9445-4412-b86b-05110f21502f") : secret "data-science-collector-collector-headless-tls" not found Oct 31 00:48:11.580728 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:48:11.580705 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c98d27b2901aaa9d0b4140c9b059347cb35a529c3f569b9d4251d6bc6679ff26\": container with ID starting with c98d27b2901aaa9d0b4140c9b059347cb35a529c3f569b9d4251d6bc6679ff26 not found: ID does not exist" containerID="c98d27b2901aaa9d0b4140c9b059347cb35a529c3f569b9d4251d6bc6679ff26" Oct 31 00:48:11.580801 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.580735 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c98d27b2901aaa9d0b4140c9b059347cb35a529c3f569b9d4251d6bc6679ff26"} err="failed to get container status \"c98d27b2901aaa9d0b4140c9b059347cb35a529c3f569b9d4251d6bc6679ff26\": rpc error: code = NotFound desc = could not find container \"c98d27b2901aaa9d0b4140c9b059347cb35a529c3f569b9d4251d6bc6679ff26\": container with ID starting with c98d27b2901aaa9d0b4140c9b059347cb35a529c3f569b9d4251d6bc6679ff26 not found: ID does not exist" Oct 31 00:48:11.580801 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.580752 5274 scope.go:122] "RemoveContainer" containerID="d8edd303852fca685538a22d0e5591290ab6c908e9637d9f9a8994a9ac9a137a" Oct 31 00:48:11.581035 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:48:11.581018 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8edd303852fca685538a22d0e5591290ab6c908e9637d9f9a8994a9ac9a137a\": container with ID starting with d8edd303852fca685538a22d0e5591290ab6c908e9637d9f9a8994a9ac9a137a not found: ID does not exist" containerID="d8edd303852fca685538a22d0e5591290ab6c908e9637d9f9a8994a9ac9a137a" Oct 31 00:48:11.581121 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.581042 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8edd303852fca685538a22d0e5591290ab6c908e9637d9f9a8994a9ac9a137a"} err="failed to get container status \"d8edd303852fca685538a22d0e5591290ab6c908e9637d9f9a8994a9ac9a137a\": rpc error: code = NotFound desc = could not find container \"d8edd303852fca685538a22d0e5591290ab6c908e9637d9f9a8994a9ac9a137a\": container with ID starting with d8edd303852fca685538a22d0e5591290ab6c908e9637d9f9a8994a9ac9a137a not found: ID does not exist" Oct 31 00:48:11.581121 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.581064 5274 scope.go:122] "RemoveContainer" containerID="a4e42cbcf4e243ecb4bbc1f7b56868a7480cb6716487d7b38b40809cca6baeef" Oct 31 00:48:11.581527 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:48:11.581501 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4e42cbcf4e243ecb4bbc1f7b56868a7480cb6716487d7b38b40809cca6baeef\": container with ID starting with a4e42cbcf4e243ecb4bbc1f7b56868a7480cb6716487d7b38b40809cca6baeef not found: ID does not exist" containerID="a4e42cbcf4e243ecb4bbc1f7b56868a7480cb6716487d7b38b40809cca6baeef" Oct 31 00:48:11.581681 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.581662 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4e42cbcf4e243ecb4bbc1f7b56868a7480cb6716487d7b38b40809cca6baeef"} err="failed to get container status \"a4e42cbcf4e243ecb4bbc1f7b56868a7480cb6716487d7b38b40809cca6baeef\": rpc error: code = NotFound desc = could not find container \"a4e42cbcf4e243ecb4bbc1f7b56868a7480cb6716487d7b38b40809cca6baeef\": container with ID starting with a4e42cbcf4e243ecb4bbc1f7b56868a7480cb6716487d7b38b40809cca6baeef not found: ID does not exist" Oct 31 00:48:11.581776 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.581683 5274 scope.go:122] "RemoveContainer" containerID="c98d27b2901aaa9d0b4140c9b059347cb35a529c3f569b9d4251d6bc6679ff26" Oct 31 00:48:11.581776 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.581502 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.581488364 podStartE2EDuration="2.581488364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:11.579739899 +0000 UTC m=+2928.216558641" watchObservedRunningTime="2026-10-31 00:48:11.581488364 +0000 UTC m=+2928.218307080" Oct 31 00:48:11.581994 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.581942 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c98d27b2901aaa9d0b4140c9b059347cb35a529c3f569b9d4251d6bc6679ff26"} err="failed to get container status \"c98d27b2901aaa9d0b4140c9b059347cb35a529c3f569b9d4251d6bc6679ff26\": rpc error: code = NotFound desc = could not find container \"c98d27b2901aaa9d0b4140c9b059347cb35a529c3f569b9d4251d6bc6679ff26\": container with ID starting with c98d27b2901aaa9d0b4140c9b059347cb35a529c3f569b9d4251d6bc6679ff26 not found: ID does not exist" Oct 31 00:48:11.582096 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.581995 5274 scope.go:122] "RemoveContainer" containerID="d8edd303852fca685538a22d0e5591290ab6c908e9637d9f9a8994a9ac9a137a" Oct 31 00:48:11.582297 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.582275 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8edd303852fca685538a22d0e5591290ab6c908e9637d9f9a8994a9ac9a137a"} err="failed to get container status \"d8edd303852fca685538a22d0e5591290ab6c908e9637d9f9a8994a9ac9a137a\": rpc error: code = NotFound desc = could not find container \"d8edd303852fca685538a22d0e5591290ab6c908e9637d9f9a8994a9ac9a137a\": container with ID starting with d8edd303852fca685538a22d0e5591290ab6c908e9637d9f9a8994a9ac9a137a not found: ID does not exist" Oct 31 00:48:11.582400 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.582298 5274 scope.go:122] "RemoveContainer" containerID="a4e42cbcf4e243ecb4bbc1f7b56868a7480cb6716487d7b38b40809cca6baeef" Oct 31 00:48:11.582616 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.582589 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4e42cbcf4e243ecb4bbc1f7b56868a7480cb6716487d7b38b40809cca6baeef"} err="failed to get container status \"a4e42cbcf4e243ecb4bbc1f7b56868a7480cb6716487d7b38b40809cca6baeef\": rpc error: code = NotFound desc = could not find container \"a4e42cbcf4e243ecb4bbc1f7b56868a7480cb6716487d7b38b40809cca6baeef\": container with ID starting with a4e42cbcf4e243ecb4bbc1f7b56868a7480cb6716487d7b38b40809cca6baeef not found: ID does not exist" Oct 31 00:48:11.582616 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.582614 5274 scope.go:122] "RemoveContainer" containerID="c081728fea9b28106665289c08489a5b4fea86ec128f6f918ba0a133fcac70ca" Oct 31 00:48:11.590137 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.590109 5274 scope.go:122] "RemoveContainer" containerID="f35f54813c05fc36e01db5a5f10e97a70f7ce37a8899f008a53c3767004040e8" Oct 31 00:48:11.597440 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.597228 5274 scope.go:122] "RemoveContainer" containerID="ae640f32786003e41921cfa4d825f2c0aa1ce2fcfcdce402db8608cc541b927f" Oct 31 00:48:11.599555 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.599539 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:48:11.605540 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.605519 5274 scope.go:122] "RemoveContainer" containerID="f574ca9f8f59b1037d24e660a67fd9c9b47f492de20b7de9bb0186c4ad87ecf6" Oct 31 00:48:11.606703 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.606686 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:48:11.613000 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.612982 5274 scope.go:122] "RemoveContainer" containerID="3e73958ad1f0f3dfb20668930575bc70cc034d6edc95fb6b63d76438d8d1a640" Oct 31 00:48:11.618662 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.618639 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-lffkt"] Oct 31 00:48:11.620560 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.620546 5274 scope.go:122] "RemoveContainer" containerID="f35f54813c05fc36e01db5a5f10e97a70f7ce37a8899f008a53c3767004040e8" Oct 31 00:48:11.620806 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:48:11.620788 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f35f54813c05fc36e01db5a5f10e97a70f7ce37a8899f008a53c3767004040e8\": container with ID starting with f35f54813c05fc36e01db5a5f10e97a70f7ce37a8899f008a53c3767004040e8 not found: ID does not exist" containerID="f35f54813c05fc36e01db5a5f10e97a70f7ce37a8899f008a53c3767004040e8" Oct 31 00:48:11.620866 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.620812 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f35f54813c05fc36e01db5a5f10e97a70f7ce37a8899f008a53c3767004040e8"} err="failed to get container status \"f35f54813c05fc36e01db5a5f10e97a70f7ce37a8899f008a53c3767004040e8\": rpc error: code = NotFound desc = could not find container \"f35f54813c05fc36e01db5a5f10e97a70f7ce37a8899f008a53c3767004040e8\": container with ID starting with f35f54813c05fc36e01db5a5f10e97a70f7ce37a8899f008a53c3767004040e8 not found: ID does not exist" Oct 31 00:48:11.620866 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.620827 5274 scope.go:122] "RemoveContainer" containerID="ae640f32786003e41921cfa4d825f2c0aa1ce2fcfcdce402db8608cc541b927f" Oct 31 00:48:11.621084 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:48:11.621064 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae640f32786003e41921cfa4d825f2c0aa1ce2fcfcdce402db8608cc541b927f\": container with ID starting with ae640f32786003e41921cfa4d825f2c0aa1ce2fcfcdce402db8608cc541b927f not found: ID does not exist" containerID="ae640f32786003e41921cfa4d825f2c0aa1ce2fcfcdce402db8608cc541b927f" Oct 31 00:48:11.621153 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.621092 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae640f32786003e41921cfa4d825f2c0aa1ce2fcfcdce402db8608cc541b927f"} err="failed to get container status \"ae640f32786003e41921cfa4d825f2c0aa1ce2fcfcdce402db8608cc541b927f\": rpc error: code = NotFound desc = could not find container \"ae640f32786003e41921cfa4d825f2c0aa1ce2fcfcdce402db8608cc541b927f\": container with ID starting with ae640f32786003e41921cfa4d825f2c0aa1ce2fcfcdce402db8608cc541b927f not found: ID does not exist" Oct 31 00:48:11.621153 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.621113 5274 scope.go:122] "RemoveContainer" containerID="f574ca9f8f59b1037d24e660a67fd9c9b47f492de20b7de9bb0186c4ad87ecf6" Oct 31 00:48:11.621342 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:48:11.621326 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f574ca9f8f59b1037d24e660a67fd9c9b47f492de20b7de9bb0186c4ad87ecf6\": container with ID starting with f574ca9f8f59b1037d24e660a67fd9c9b47f492de20b7de9bb0186c4ad87ecf6 not found: ID does not exist" containerID="f574ca9f8f59b1037d24e660a67fd9c9b47f492de20b7de9bb0186c4ad87ecf6" Oct 31 00:48:11.621398 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.621345 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f574ca9f8f59b1037d24e660a67fd9c9b47f492de20b7de9bb0186c4ad87ecf6"} err="failed to get container status \"f574ca9f8f59b1037d24e660a67fd9c9b47f492de20b7de9bb0186c4ad87ecf6\": rpc error: code = NotFound desc = could not find container \"f574ca9f8f59b1037d24e660a67fd9c9b47f492de20b7de9bb0186c4ad87ecf6\": container with ID starting with f574ca9f8f59b1037d24e660a67fd9c9b47f492de20b7de9bb0186c4ad87ecf6 not found: ID does not exist" Oct 31 00:48:11.621398 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.621357 5274 scope.go:122] "RemoveContainer" containerID="3e73958ad1f0f3dfb20668930575bc70cc034d6edc95fb6b63d76438d8d1a640" Oct 31 00:48:11.621587 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:48:11.621571 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e73958ad1f0f3dfb20668930575bc70cc034d6edc95fb6b63d76438d8d1a640\": container with ID starting with 3e73958ad1f0f3dfb20668930575bc70cc034d6edc95fb6b63d76438d8d1a640 not found: ID does not exist" containerID="3e73958ad1f0f3dfb20668930575bc70cc034d6edc95fb6b63d76438d8d1a640" Oct 31 00:48:11.621630 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.621590 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e73958ad1f0f3dfb20668930575bc70cc034d6edc95fb6b63d76438d8d1a640"} err="failed to get container status \"3e73958ad1f0f3dfb20668930575bc70cc034d6edc95fb6b63d76438d8d1a640\": rpc error: code = NotFound desc = could not find container \"3e73958ad1f0f3dfb20668930575bc70cc034d6edc95fb6b63d76438d8d1a640\": container with ID starting with 3e73958ad1f0f3dfb20668930575bc70cc034d6edc95fb6b63d76438d8d1a640 not found: ID does not exist" Oct 31 00:48:11.621630 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.621601 5274 scope.go:122] "RemoveContainer" containerID="f35f54813c05fc36e01db5a5f10e97a70f7ce37a8899f008a53c3767004040e8" Oct 31 00:48:11.621765 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.621751 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f35f54813c05fc36e01db5a5f10e97a70f7ce37a8899f008a53c3767004040e8"} err="failed to get container status \"f35f54813c05fc36e01db5a5f10e97a70f7ce37a8899f008a53c3767004040e8\": rpc error: code = NotFound desc = could not find container \"f35f54813c05fc36e01db5a5f10e97a70f7ce37a8899f008a53c3767004040e8\": container with ID starting with f35f54813c05fc36e01db5a5f10e97a70f7ce37a8899f008a53c3767004040e8 not found: ID does not exist" Oct 31 00:48:11.621809 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.621766 5274 scope.go:122] "RemoveContainer" containerID="ae640f32786003e41921cfa4d825f2c0aa1ce2fcfcdce402db8608cc541b927f" Oct 31 00:48:11.622088 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.622071 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae640f32786003e41921cfa4d825f2c0aa1ce2fcfcdce402db8608cc541b927f"} err="failed to get container status \"ae640f32786003e41921cfa4d825f2c0aa1ce2fcfcdce402db8608cc541b927f\": rpc error: code = NotFound desc = could not find container \"ae640f32786003e41921cfa4d825f2c0aa1ce2fcfcdce402db8608cc541b927f\": container with ID starting with ae640f32786003e41921cfa4d825f2c0aa1ce2fcfcdce402db8608cc541b927f not found: ID does not exist" Oct 31 00:48:11.622136 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.622089 5274 scope.go:122] "RemoveContainer" containerID="f574ca9f8f59b1037d24e660a67fd9c9b47f492de20b7de9bb0186c4ad87ecf6" Oct 31 00:48:11.622291 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.622276 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f574ca9f8f59b1037d24e660a67fd9c9b47f492de20b7de9bb0186c4ad87ecf6"} err="failed to get container status \"f574ca9f8f59b1037d24e660a67fd9c9b47f492de20b7de9bb0186c4ad87ecf6\": rpc error: code = NotFound desc = could not find container \"f574ca9f8f59b1037d24e660a67fd9c9b47f492de20b7de9bb0186c4ad87ecf6\": container with ID starting with f574ca9f8f59b1037d24e660a67fd9c9b47f492de20b7de9bb0186c4ad87ecf6 not found: ID does not exist" Oct 31 00:48:11.622342 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.622291 5274 scope.go:122] "RemoveContainer" containerID="3e73958ad1f0f3dfb20668930575bc70cc034d6edc95fb6b63d76438d8d1a640" Oct 31 00:48:11.622553 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.622514 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e73958ad1f0f3dfb20668930575bc70cc034d6edc95fb6b63d76438d8d1a640"} err="failed to get container status \"3e73958ad1f0f3dfb20668930575bc70cc034d6edc95fb6b63d76438d8d1a640\": rpc error: code = NotFound desc = could not find container \"3e73958ad1f0f3dfb20668930575bc70cc034d6edc95fb6b63d76438d8d1a640\": container with ID starting with 3e73958ad1f0f3dfb20668930575bc70cc034d6edc95fb6b63d76438d8d1a640 not found: ID does not exist" Oct 31 00:48:11.622650 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.622555 5274 scope.go:122] "RemoveContainer" containerID="f35f54813c05fc36e01db5a5f10e97a70f7ce37a8899f008a53c3767004040e8" Oct 31 00:48:11.622866 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.622844 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f35f54813c05fc36e01db5a5f10e97a70f7ce37a8899f008a53c3767004040e8"} err="failed to get container status \"f35f54813c05fc36e01db5a5f10e97a70f7ce37a8899f008a53c3767004040e8\": rpc error: code = NotFound desc = could not find container \"f35f54813c05fc36e01db5a5f10e97a70f7ce37a8899f008a53c3767004040e8\": container with ID starting with f35f54813c05fc36e01db5a5f10e97a70f7ce37a8899f008a53c3767004040e8 not found: ID does not exist" Oct 31 00:48:11.622866 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.622867 5274 scope.go:122] "RemoveContainer" containerID="ae640f32786003e41921cfa4d825f2c0aa1ce2fcfcdce402db8608cc541b927f" Oct 31 00:48:11.623333 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.623299 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae640f32786003e41921cfa4d825f2c0aa1ce2fcfcdce402db8608cc541b927f"} err="failed to get container status \"ae640f32786003e41921cfa4d825f2c0aa1ce2fcfcdce402db8608cc541b927f\": rpc error: code = NotFound desc = could not find container \"ae640f32786003e41921cfa4d825f2c0aa1ce2fcfcdce402db8608cc541b927f\": container with ID starting with ae640f32786003e41921cfa4d825f2c0aa1ce2fcfcdce402db8608cc541b927f not found: ID does not exist" Oct 31 00:48:11.623333 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.623333 5274 scope.go:122] "RemoveContainer" containerID="f574ca9f8f59b1037d24e660a67fd9c9b47f492de20b7de9bb0186c4ad87ecf6" Oct 31 00:48:11.624020 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.623993 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f574ca9f8f59b1037d24e660a67fd9c9b47f492de20b7de9bb0186c4ad87ecf6"} err="failed to get container status \"f574ca9f8f59b1037d24e660a67fd9c9b47f492de20b7de9bb0186c4ad87ecf6\": rpc error: code = NotFound desc = could not find container \"f574ca9f8f59b1037d24e660a67fd9c9b47f492de20b7de9bb0186c4ad87ecf6\": container with ID starting with f574ca9f8f59b1037d24e660a67fd9c9b47f492de20b7de9bb0186c4ad87ecf6 not found: ID does not exist" Oct 31 00:48:11.624020 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.624020 5274 scope.go:122] "RemoveContainer" containerID="3e73958ad1f0f3dfb20668930575bc70cc034d6edc95fb6b63d76438d8d1a640" Oct 31 00:48:11.624286 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.624268 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e73958ad1f0f3dfb20668930575bc70cc034d6edc95fb6b63d76438d8d1a640"} err="failed to get container status \"3e73958ad1f0f3dfb20668930575bc70cc034d6edc95fb6b63d76438d8d1a640\": rpc error: code = NotFound desc = could not find container \"3e73958ad1f0f3dfb20668930575bc70cc034d6edc95fb6b63d76438d8d1a640\": container with ID starting with 3e73958ad1f0f3dfb20668930575bc70cc034d6edc95fb6b63d76438d8d1a640 not found: ID does not exist" Oct 31 00:48:11.624858 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.624843 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-lffkt"] Oct 31 00:48:11.639266 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.639243 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:48:11.644283 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:11.644266 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:48:12.083706 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:48:12.083673 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-monitoring-tls: secret "data-science-collector-monitoring-tls" not found Oct 31 00:48:12.083706 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:48:12.083693 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-tls: secret "data-science-collector-tls" not found Oct 31 00:48:12.083706 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:48:12.083702 5274 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-collector-collector-a0f9182a: configmap "data-science-collector-collector-a0f9182a" not found Oct 31 00:48:12.083950 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:48:12.083693 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found Oct 31 00:48:12.083950 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:48:12.083755 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-monitoring-tls-certs podName:b58466ba-9445-4412-b86b-05110f21502f nodeName:}" failed. No retries permitted until 2026-10-31 00:48:13.083738407 +0000 UTC m=+2929.720557100 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "monitoring-tls-certs" (UniqueName: "kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-monitoring-tls-certs") pod "data-science-collector-collector-0" (UID: "b58466ba-9445-4412-b86b-05110f21502f") : secret "data-science-collector-monitoring-tls" not found Oct 31 00:48:12.083950 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:48:12.083768 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-tls-certs podName:b58466ba-9445-4412-b86b-05110f21502f nodeName:}" failed. No retries permitted until 2026-10-31 00:48:13.083762642 +0000 UTC m=+2929.720581335 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-tls-certs") pod "data-science-collector-collector-0" (UID: "b58466ba-9445-4412-b86b-05110f21502f") : secret "data-science-collector-tls" not found Oct 31 00:48:12.083950 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:48:12.083777 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b58466ba-9445-4412-b86b-05110f21502f-otc-internal podName:b58466ba-9445-4412-b86b-05110f21502f nodeName:}" failed. No retries permitted until 2026-10-31 00:48:13.083772661 +0000 UTC m=+2929.720591354 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "otc-internal" (UniqueName: "kubernetes.io/configmap/b58466ba-9445-4412-b86b-05110f21502f-otc-internal") pod "data-science-collector-collector-0" (UID: "b58466ba-9445-4412-b86b-05110f21502f") : configmap "data-science-collector-collector-a0f9182a" not found Oct 31 00:48:12.083950 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:48:12.083788 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-otlp-tls-certs podName:b58466ba-9445-4412-b86b-05110f21502f nodeName:}" failed. No retries permitted until 2026-10-31 00:48:13.083783283 +0000 UTC m=+2929.720601976 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-otlp-tls-certs") pod "data-science-collector-collector-0" (UID: "b58466ba-9445-4412-b86b-05110f21502f") : secret "data-science-collector-collector-headless-tls" not found Oct 31 00:48:12.566232 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:12.566199 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="b58466ba-9445-4412-b86b-05110f21502f" containerName="otc-container" containerID="cri-o://25a69517316026d02a918798b389e3ad66bd25d2216f9c4644e347ab5cdfcb89" gracePeriod=30 Oct 31 00:48:12.813990 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:12.813945 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:12.890006 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:12.889985 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b58466ba-9445-4412-b86b-05110f21502f-otc-internal\" (UniqueName: \"kubernetes.io/configmap/b58466ba-9445-4412-b86b-05110f21502f-otc-internal\") pod \"b58466ba-9445-4412-b86b-05110f21502f\" (UID: \"b58466ba-9445-4412-b86b-05110f21502f\") " Oct 31 00:48:12.890132 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:12.890046 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-tls-certs\" (UniqueName: \"kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-tls-certs\") pod \"b58466ba-9445-4412-b86b-05110f21502f\" (UID: \"b58466ba-9445-4412-b86b-05110f21502f\") " Oct 31 00:48:12.890132 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:12.890092 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b58466ba-9445-4412-b86b-05110f21502f-kube-api-access-d4g8t\" (UniqueName: \"kubernetes.io/projected/b58466ba-9445-4412-b86b-05110f21502f-kube-api-access-d4g8t\") pod \"b58466ba-9445-4412-b86b-05110f21502f\" (UID: \"b58466ba-9445-4412-b86b-05110f21502f\") " Oct 31 00:48:12.890132 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:12.890110 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-data-science-collector-ta-client-cert\") pod \"b58466ba-9445-4412-b86b-05110f21502f\" (UID: \"b58466ba-9445-4412-b86b-05110f21502f\") " Oct 31 00:48:12.890309 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:12.890155 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-monitoring-tls-certs\") pod \"b58466ba-9445-4412-b86b-05110f21502f\" (UID: \"b58466ba-9445-4412-b86b-05110f21502f\") " Oct 31 00:48:12.890309 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:12.890246 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-otlp-tls-certs\") pod \"b58466ba-9445-4412-b86b-05110f21502f\" (UID: \"b58466ba-9445-4412-b86b-05110f21502f\") " Oct 31 00:48:12.890431 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:12.890382 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b58466ba-9445-4412-b86b-05110f21502f-otc-internal" pod "b58466ba-9445-4412-b86b-05110f21502f" (UID: "b58466ba-9445-4412-b86b-05110f21502f"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:12.890495 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:12.890456 5274 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b58466ba-9445-4412-b86b-05110f21502f-otc-internal\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:12.892235 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:12.892204 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-data-science-collector-ta-client-cert" pod "b58466ba-9445-4412-b86b-05110f21502f" (UID: "b58466ba-9445-4412-b86b-05110f21502f"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:12.892360 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:12.892303 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b58466ba-9445-4412-b86b-05110f21502f-kube-api-access-d4g8t" pod "b58466ba-9445-4412-b86b-05110f21502f" (UID: "b58466ba-9445-4412-b86b-05110f21502f"). InnerVolumeSpecName "kube-api-access-d4g8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:12.892412 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:12.892364 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-tls-certs" pod "b58466ba-9445-4412-b86b-05110f21502f" (UID: "b58466ba-9445-4412-b86b-05110f21502f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:12.892583 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:12.892566 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-otlp-tls-certs" pod "b58466ba-9445-4412-b86b-05110f21502f" (UID: "b58466ba-9445-4412-b86b-05110f21502f"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:12.892637 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:12.892578 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-monitoring-tls-certs" pod "b58466ba-9445-4412-b86b-05110f21502f" (UID: "b58466ba-9445-4412-b86b-05110f21502f"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:12.990927 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:12.990902 5274 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-otlp-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:12.990927 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:12.990924 5274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:12.991101 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:12.990941 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4g8t\" (UniqueName: \"kubernetes.io/projected/b58466ba-9445-4412-b86b-05110f21502f-kube-api-access-d4g8t\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:12.991101 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:12.990976 5274 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-data-science-collector-ta-client-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:12.991101 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:12.990990 5274 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b58466ba-9445-4412-b86b-05110f21502f-monitoring-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:13.481717 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:13.481681 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e9aed47-75bf-4ce4-ad8e-21c73da85043" path="/var/lib/kubelet/pods/2e9aed47-75bf-4ce4-ad8e-21c73da85043/volumes" Oct 31 00:48:13.482238 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:13.482224 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46200120-336e-49c2-81f2-fc7a540ca5df" path="/var/lib/kubelet/pods/46200120-336e-49c2-81f2-fc7a540ca5df/volumes" Oct 31 00:48:13.482828 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:13.482813 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea3bb993-f20b-4a7a-a9dc-f198aafd9a28" path="/var/lib/kubelet/pods/ea3bb993-f20b-4a7a-a9dc-f198aafd9a28/volumes" Oct 31 00:48:13.571663 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:13.571638 5274 generic.go:356] "Generic (PLEG): container finished" podID="b58466ba-9445-4412-b86b-05110f21502f" containerID="25a69517316026d02a918798b389e3ad66bd25d2216f9c4644e347ab5cdfcb89" exitCode=2 Oct 31 00:48:13.572079 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:13.571686 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:13.572079 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:13.571728 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"b58466ba-9445-4412-b86b-05110f21502f","Type":"ContainerDied","Data":"25a69517316026d02a918798b389e3ad66bd25d2216f9c4644e347ab5cdfcb89"} Oct 31 00:48:13.572079 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:13.571783 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"b58466ba-9445-4412-b86b-05110f21502f","Type":"ContainerDied","Data":"5ec6c24b25ea071f13624420dc77eeb2d6bb9acf2f4ff0c6a15874d8ae82b227"} Oct 31 00:48:13.572079 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:13.571812 5274 scope.go:122] "RemoveContainer" containerID="25a69517316026d02a918798b389e3ad66bd25d2216f9c4644e347ab5cdfcb89" Oct 31 00:48:13.598119 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:13.598096 5274 scope.go:122] "RemoveContainer" containerID="25a69517316026d02a918798b389e3ad66bd25d2216f9c4644e347ab5cdfcb89" Oct 31 00:48:13.598400 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:48:13.598380 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25a69517316026d02a918798b389e3ad66bd25d2216f9c4644e347ab5cdfcb89\": container with ID starting with 25a69517316026d02a918798b389e3ad66bd25d2216f9c4644e347ab5cdfcb89 not found: ID does not exist" containerID="25a69517316026d02a918798b389e3ad66bd25d2216f9c4644e347ab5cdfcb89" Oct 31 00:48:13.598482 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:13.598409 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25a69517316026d02a918798b389e3ad66bd25d2216f9c4644e347ab5cdfcb89"} err="failed to get container status \"25a69517316026d02a918798b389e3ad66bd25d2216f9c4644e347ab5cdfcb89\": rpc error: code = NotFound desc = could not find container \"25a69517316026d02a918798b389e3ad66bd25d2216f9c4644e347ab5cdfcb89\": container with ID starting with 25a69517316026d02a918798b389e3ad66bd25d2216f9c4644e347ab5cdfcb89 not found: ID does not exist" Oct 31 00:48:13.609228 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:13.609205 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:48:13.613467 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:13.613448 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:48:14.310317 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:14.310280 5274 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="ea3bb993-f20b-4a7a-a9dc-f198aafd9a28" containerName="alertmanager" probeResult="failure" output="Get \"http://10.134.0.126:9093/-/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:15.483495 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:15.483465 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b58466ba-9445-4412-b86b-05110f21502f" path="/var/lib/kubelet/pods/b58466ba-9445-4412-b86b-05110f21502f/volumes" Oct 31 00:48:18.374709 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.374673 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-bdcsb"] Oct 31 00:48:18.375106 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.374998 5274 cpu_manager.go:414] "containerMap: removing container" podUID="ea3bb993-f20b-4a7a-a9dc-f198aafd9a28" containerName="init-config-reloader" Oct 31 00:48:18.375106 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.375011 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea3bb993-f20b-4a7a-a9dc-f198aafd9a28" containerName="init-config-reloader" Oct 31 00:48:18.375106 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.375027 5274 cpu_manager.go:414] "containerMap: removing container" podUID="2e9aed47-75bf-4ce4-ad8e-21c73da85043" containerName="init-config-reloader" Oct 31 00:48:18.375106 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.375032 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e9aed47-75bf-4ce4-ad8e-21c73da85043" containerName="init-config-reloader" Oct 31 00:48:18.375106 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.375040 5274 cpu_manager.go:414] "containerMap: removing container" podUID="2e9aed47-75bf-4ce4-ad8e-21c73da85043" containerName="thanos-sidecar" Oct 31 00:48:18.375106 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.375046 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e9aed47-75bf-4ce4-ad8e-21c73da85043" containerName="thanos-sidecar" Oct 31 00:48:18.375106 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.375054 5274 cpu_manager.go:414] "containerMap: removing container" podUID="ea3bb993-f20b-4a7a-a9dc-f198aafd9a28" containerName="alertmanager" Oct 31 00:48:18.375106 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.375059 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea3bb993-f20b-4a7a-a9dc-f198aafd9a28" containerName="alertmanager" Oct 31 00:48:18.375106 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.375065 5274 cpu_manager.go:414] "containerMap: removing container" podUID="ea3bb993-f20b-4a7a-a9dc-f198aafd9a28" containerName="config-reloader" Oct 31 00:48:18.375106 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.375069 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea3bb993-f20b-4a7a-a9dc-f198aafd9a28" containerName="config-reloader" Oct 31 00:48:18.375106 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.375076 5274 cpu_manager.go:414] "containerMap: removing container" podUID="46200120-336e-49c2-81f2-fc7a540ca5df" containerName="korrel8r" Oct 31 00:48:18.375106 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.375080 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46200120-336e-49c2-81f2-fc7a540ca5df" containerName="korrel8r" Oct 31 00:48:18.375106 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.375086 5274 cpu_manager.go:414] "containerMap: removing container" podUID="2e9aed47-75bf-4ce4-ad8e-21c73da85043" containerName="prometheus" Oct 31 00:48:18.375106 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.375091 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e9aed47-75bf-4ce4-ad8e-21c73da85043" containerName="prometheus" Oct 31 00:48:18.375106 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.375096 5274 cpu_manager.go:414] "containerMap: removing container" podUID="2e9aed47-75bf-4ce4-ad8e-21c73da85043" containerName="config-reloader" Oct 31 00:48:18.375106 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.375100 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e9aed47-75bf-4ce4-ad8e-21c73da85043" containerName="config-reloader" Oct 31 00:48:18.375569 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.375119 5274 cpu_manager.go:414] "containerMap: removing container" podUID="b58466ba-9445-4412-b86b-05110f21502f" containerName="otc-container" Oct 31 00:48:18.375569 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.375123 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b58466ba-9445-4412-b86b-05110f21502f" containerName="otc-container" Oct 31 00:48:18.375569 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.375173 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="b58466ba-9445-4412-b86b-05110f21502f" containerName="otc-container" Oct 31 00:48:18.375569 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.375184 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e9aed47-75bf-4ce4-ad8e-21c73da85043" containerName="thanos-sidecar" Oct 31 00:48:18.375569 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.375190 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea3bb993-f20b-4a7a-a9dc-f198aafd9a28" containerName="alertmanager" Oct 31 00:48:18.375569 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.375196 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea3bb993-f20b-4a7a-a9dc-f198aafd9a28" containerName="config-reloader" Oct 31 00:48:18.375569 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.375202 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="46200120-336e-49c2-81f2-fc7a540ca5df" containerName="korrel8r" Oct 31 00:48:18.375569 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.375207 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e9aed47-75bf-4ce4-ad8e-21c73da85043" containerName="prometheus" Oct 31 00:48:18.375569 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.375213 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e9aed47-75bf-4ce4-ad8e-21c73da85043" containerName="config-reloader" Oct 31 00:48:18.379498 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.379480 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-bdcsb" Oct 31 00:48:18.388309 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.388285 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\"" Oct 31 00:48:18.388888 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.388868 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:48:18.389007 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.388991 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:48:18.389103 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.389088 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\"" Oct 31 00:48:18.389154 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.389136 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 31 00:48:18.389258 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.389244 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-sctk9\"" Oct 31 00:48:18.389312 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.389256 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-bdcsb"] Oct 31 00:48:18.389349 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.389316 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:48:18.430218 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.430192 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45d15e31-0f2f-4c64-b35b-c133895ccea5-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-bdcsb\" (UID: \"45d15e31-0f2f-4c64-b35b-c133895ccea5\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-bdcsb" Oct 31 00:48:18.430334 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.430232 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/45d15e31-0f2f-4c64-b35b-c133895ccea5-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-bdcsb\" (UID: \"45d15e31-0f2f-4c64-b35b-c133895ccea5\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-bdcsb" Oct 31 00:48:18.430334 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.430257 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jlt6\" (UniqueName: \"kubernetes.io/projected/45d15e31-0f2f-4c64-b35b-c133895ccea5-kube-api-access-4jlt6\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-bdcsb\" (UID: \"45d15e31-0f2f-4c64-b35b-c133895ccea5\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-bdcsb" Oct 31 00:48:18.430428 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.430327 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/45d15e31-0f2f-4c64-b35b-c133895ccea5-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-bdcsb\" (UID: \"45d15e31-0f2f-4c64-b35b-c133895ccea5\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-bdcsb" Oct 31 00:48:18.430428 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.430364 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/45d15e31-0f2f-4c64-b35b-c133895ccea5-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-bdcsb\" (UID: \"45d15e31-0f2f-4c64-b35b-c133895ccea5\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-bdcsb" Oct 31 00:48:18.433671 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.433650 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-c8mg8"] Oct 31 00:48:18.437003 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.436989 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c8mg8" Oct 31 00:48:18.449492 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.449469 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-c8mg8"] Oct 31 00:48:18.462706 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.462682 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-97s5k\"" Oct 31 00:48:18.462706 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.462685 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 31 00:48:18.487109 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.487089 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 31 00:48:18.531532 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.531509 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/45d15e31-0f2f-4c64-b35b-c133895ccea5-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-bdcsb\" (UID: \"45d15e31-0f2f-4c64-b35b-c133895ccea5\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-bdcsb" Oct 31 00:48:18.531617 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.531539 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jlt6\" (UniqueName: \"kubernetes.io/projected/45d15e31-0f2f-4c64-b35b-c133895ccea5-kube-api-access-4jlt6\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-bdcsb\" (UID: \"45d15e31-0f2f-4c64-b35b-c133895ccea5\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-bdcsb" Oct 31 00:48:18.531617 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.531562 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/45d15e31-0f2f-4c64-b35b-c133895ccea5-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-bdcsb\" (UID: \"45d15e31-0f2f-4c64-b35b-c133895ccea5\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-bdcsb" Oct 31 00:48:18.531617 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.531591 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e662ea68-3516-4efd-8e66-f86532b11225-tls\") pod \"korrel8r-5cbcc89b8f-c8mg8\" (UID: \"e662ea68-3516-4efd-8e66-f86532b11225\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c8mg8" Oct 31 00:48:18.531772 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.531617 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/45d15e31-0f2f-4c64-b35b-c133895ccea5-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-bdcsb\" (UID: \"45d15e31-0f2f-4c64-b35b-c133895ccea5\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-bdcsb" Oct 31 00:48:18.531772 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:48:18.531644 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found Oct 31 00:48:18.532267 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:48:18.532234 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45d15e31-0f2f-4c64-b35b-c133895ccea5-prometheus-client-cert podName:45d15e31-0f2f-4c64-b35b-c133895ccea5 nodeName:}" failed. No retries permitted until 2026-10-31 00:48:19.031698054 +0000 UTC m=+2935.668516759 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/45d15e31-0f2f-4c64-b35b-c133895ccea5-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-bdcsb" (UID: "45d15e31-0f2f-4c64-b35b-c133895ccea5") : secret "prometheus-operated-tls" not found Oct 31 00:48:18.532372 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.532336 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e662ea68-3516-4efd-8e66-f86532b11225-config\") pod \"korrel8r-5cbcc89b8f-c8mg8\" (UID: \"e662ea68-3516-4efd-8e66-f86532b11225\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c8mg8" Oct 31 00:48:18.532434 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.532406 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wzs5\" (UniqueName: \"kubernetes.io/projected/e662ea68-3516-4efd-8e66-f86532b11225-kube-api-access-8wzs5\") pod \"korrel8r-5cbcc89b8f-c8mg8\" (UID: \"e662ea68-3516-4efd-8e66-f86532b11225\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c8mg8" Oct 31 00:48:18.532487 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.532466 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45d15e31-0f2f-4c64-b35b-c133895ccea5-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-bdcsb\" (UID: \"45d15e31-0f2f-4c64-b35b-c133895ccea5\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-bdcsb" Oct 31 00:48:18.537970 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.533078 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/45d15e31-0f2f-4c64-b35b-c133895ccea5-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-bdcsb\" (UID: \"45d15e31-0f2f-4c64-b35b-c133895ccea5\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-bdcsb" Oct 31 00:48:18.537970 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.535307 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45d15e31-0f2f-4c64-b35b-c133895ccea5-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-bdcsb\" (UID: \"45d15e31-0f2f-4c64-b35b-c133895ccea5\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-bdcsb" Oct 31 00:48:18.540316 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.540293 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/45d15e31-0f2f-4c64-b35b-c133895ccea5-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-bdcsb\" (UID: \"45d15e31-0f2f-4c64-b35b-c133895ccea5\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-bdcsb" Oct 31 00:48:18.545044 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.545021 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jlt6\" (UniqueName: \"kubernetes.io/projected/45d15e31-0f2f-4c64-b35b-c133895ccea5-kube-api-access-4jlt6\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-bdcsb\" (UID: \"45d15e31-0f2f-4c64-b35b-c133895ccea5\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-bdcsb" Oct 31 00:48:18.633768 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.633697 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wzs5\" (UniqueName: \"kubernetes.io/projected/e662ea68-3516-4efd-8e66-f86532b11225-kube-api-access-8wzs5\") pod \"korrel8r-5cbcc89b8f-c8mg8\" (UID: \"e662ea68-3516-4efd-8e66-f86532b11225\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c8mg8" Oct 31 00:48:18.633882 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.633842 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e662ea68-3516-4efd-8e66-f86532b11225-tls\") pod \"korrel8r-5cbcc89b8f-c8mg8\" (UID: \"e662ea68-3516-4efd-8e66-f86532b11225\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c8mg8" Oct 31 00:48:18.633926 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.633888 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e662ea68-3516-4efd-8e66-f86532b11225-config\") pod \"korrel8r-5cbcc89b8f-c8mg8\" (UID: \"e662ea68-3516-4efd-8e66-f86532b11225\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c8mg8" Oct 31 00:48:18.634891 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.634865 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e662ea68-3516-4efd-8e66-f86532b11225-config\") pod \"korrel8r-5cbcc89b8f-c8mg8\" (UID: \"e662ea68-3516-4efd-8e66-f86532b11225\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c8mg8" Oct 31 00:48:18.636051 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.636029 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e662ea68-3516-4efd-8e66-f86532b11225-tls\") pod \"korrel8r-5cbcc89b8f-c8mg8\" (UID: \"e662ea68-3516-4efd-8e66-f86532b11225\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c8mg8" Oct 31 00:48:18.648571 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.648552 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wzs5\" (UniqueName: \"kubernetes.io/projected/e662ea68-3516-4efd-8e66-f86532b11225-kube-api-access-8wzs5\") pod \"korrel8r-5cbcc89b8f-c8mg8\" (UID: \"e662ea68-3516-4efd-8e66-f86532b11225\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c8mg8" Oct 31 00:48:18.749781 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.749755 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c8mg8" Oct 31 00:48:18.926223 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.925816 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:48:18.930760 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.930738 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:18.936163 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.936131 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/893e420c-f593-4011-83af-12f5eeb9466a-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"893e420c-f593-4011-83af-12f5eeb9466a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:18.936295 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.936227 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzhcd\" (UniqueName: \"kubernetes.io/projected/893e420c-f593-4011-83af-12f5eeb9466a-kube-api-access-zzhcd\") pod \"data-science-collector-collector-0\" (UID: \"893e420c-f593-4011-83af-12f5eeb9466a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:18.936295 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.936290 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/893e420c-f593-4011-83af-12f5eeb9466a-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"893e420c-f593-4011-83af-12f5eeb9466a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:18.936429 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.936312 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/893e420c-f593-4011-83af-12f5eeb9466a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"893e420c-f593-4011-83af-12f5eeb9466a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:18.936429 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.936395 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/893e420c-f593-4011-83af-12f5eeb9466a-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"893e420c-f593-4011-83af-12f5eeb9466a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:18.936429 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.936425 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/893e420c-f593-4011-83af-12f5eeb9466a-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"893e420c-f593-4011-83af-12f5eeb9466a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:18.937832 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.937810 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:48:18.938277 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.938254 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:48:18.940335 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.940314 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:48:18.941747 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.941725 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:48:18.943408 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:18.943342 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:48:19.032406 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:19.032371 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-c8mg8"] Oct 31 00:48:19.035918 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:48:19.035891 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode662ea68_3516_4efd_8e66_f86532b11225.slice/crio-cfa14bb9519e8eabcb78562280f733dc31fa749bedc248b7772244a50248d526 WatchSource:0}: Error finding container cfa14bb9519e8eabcb78562280f733dc31fa749bedc248b7772244a50248d526: Status 404 returned error can't find the container with id cfa14bb9519e8eabcb78562280f733dc31fa749bedc248b7772244a50248d526 Oct 31 00:48:19.037371 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:19.037347 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzhcd\" (UniqueName: \"kubernetes.io/projected/893e420c-f593-4011-83af-12f5eeb9466a-kube-api-access-zzhcd\") pod \"data-science-collector-collector-0\" (UID: \"893e420c-f593-4011-83af-12f5eeb9466a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:19.037466 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:19.037408 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/893e420c-f593-4011-83af-12f5eeb9466a-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"893e420c-f593-4011-83af-12f5eeb9466a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:19.037466 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:19.037425 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/893e420c-f593-4011-83af-12f5eeb9466a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"893e420c-f593-4011-83af-12f5eeb9466a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:19.037466 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:19.037445 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/893e420c-f593-4011-83af-12f5eeb9466a-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"893e420c-f593-4011-83af-12f5eeb9466a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:19.037625 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:19.037469 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/893e420c-f593-4011-83af-12f5eeb9466a-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"893e420c-f593-4011-83af-12f5eeb9466a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:19.037625 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:19.037493 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/45d15e31-0f2f-4c64-b35b-c133895ccea5-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-bdcsb\" (UID: \"45d15e31-0f2f-4c64-b35b-c133895ccea5\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-bdcsb" Oct 31 00:48:19.037625 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:19.037559 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/893e420c-f593-4011-83af-12f5eeb9466a-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"893e420c-f593-4011-83af-12f5eeb9466a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:19.037775 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:48:19.037692 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found Oct 31 00:48:19.037775 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:48:19.037741 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/893e420c-f593-4011-83af-12f5eeb9466a-otlp-tls-certs podName:893e420c-f593-4011-83af-12f5eeb9466a nodeName:}" failed. No retries permitted until 2026-10-31 00:48:19.537727059 +0000 UTC m=+2936.174545753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/893e420c-f593-4011-83af-12f5eeb9466a-otlp-tls-certs") pod "data-science-collector-collector-0" (UID: "893e420c-f593-4011-83af-12f5eeb9466a") : secret "data-science-collector-collector-headless-tls" not found Oct 31 00:48:19.038385 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:19.038363 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:48:19.038500 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:19.038428 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/893e420c-f593-4011-83af-12f5eeb9466a-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"893e420c-f593-4011-83af-12f5eeb9466a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:19.041103 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:19.041083 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/893e420c-f593-4011-83af-12f5eeb9466a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"893e420c-f593-4011-83af-12f5eeb9466a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:19.041180 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:19.041133 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/893e420c-f593-4011-83af-12f5eeb9466a-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"893e420c-f593-4011-83af-12f5eeb9466a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:19.041311 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:19.041293 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/45d15e31-0f2f-4c64-b35b-c133895ccea5-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-bdcsb\" (UID: \"45d15e31-0f2f-4c64-b35b-c133895ccea5\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-bdcsb" Oct 31 00:48:19.050903 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:19.050804 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/893e420c-f593-4011-83af-12f5eeb9466a-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"893e420c-f593-4011-83af-12f5eeb9466a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:19.054320 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:19.054300 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzhcd\" (UniqueName: \"kubernetes.io/projected/893e420c-f593-4011-83af-12f5eeb9466a-kube-api-access-zzhcd\") pod \"data-science-collector-collector-0\" (UID: \"893e420c-f593-4011-83af-12f5eeb9466a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:19.294581 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:19.294503 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-bdcsb" Oct 31 00:48:19.543061 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:19.543030 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/893e420c-f593-4011-83af-12f5eeb9466a-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"893e420c-f593-4011-83af-12f5eeb9466a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:19.545506 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:19.545451 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/893e420c-f593-4011-83af-12f5eeb9466a-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"893e420c-f593-4011-83af-12f5eeb9466a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:19.548308 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:19.548281 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:19.569889 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:19.569196 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-bdcsb"] Oct 31 00:48:19.571576 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:48:19.571556 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d15e31_0f2f_4c64_b35b_c133895ccea5.slice/crio-ec78552903df1b29df579c8ae3630f2d15110392eddd08b4a21dc4d9683f408d WatchSource:0}: Error finding container ec78552903df1b29df579c8ae3630f2d15110392eddd08b4a21dc4d9683f408d: Status 404 returned error can't find the container with id ec78552903df1b29df579c8ae3630f2d15110392eddd08b4a21dc4d9683f408d Oct 31 00:48:19.607030 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:19.607001 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-bdcsb" event={"ID":"45d15e31-0f2f-4c64-b35b-c133895ccea5","Type":"ContainerStarted","Data":"ec78552903df1b29df579c8ae3630f2d15110392eddd08b4a21dc4d9683f408d"} Oct 31 00:48:19.609406 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:19.609384 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c8mg8" event={"ID":"e662ea68-3516-4efd-8e66-f86532b11225","Type":"ContainerStarted","Data":"a01c811b9b6ee96f60c8a7dac44549b08b9e66ccfa601498827d765dbbf42882"} Oct 31 00:48:19.609517 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:19.609429 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c8mg8" event={"ID":"e662ea68-3516-4efd-8e66-f86532b11225","Type":"ContainerStarted","Data":"cfa14bb9519e8eabcb78562280f733dc31fa749bedc248b7772244a50248d526"} Oct 31 00:48:19.609587 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:19.609561 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c8mg8" Oct 31 00:48:19.630817 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:19.630770 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c8mg8" podStartSLOduration=1.630755396 podStartE2EDuration="1.630755396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:19.628950484 +0000 UTC m=+2936.265769199" watchObservedRunningTime="2026-10-31 00:48:19.630755396 +0000 UTC m=+2936.267574106" Oct 31 00:48:19.828840 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:19.828731 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:48:19.831251 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:48:19.831215 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod893e420c_f593_4011_83af_12f5eeb9466a.slice/crio-853c953b7a7635e2785996c2dcd0269c5db56c25e172c4854e26356ac8a3a813 WatchSource:0}: Error finding container 853c953b7a7635e2785996c2dcd0269c5db56c25e172c4854e26356ac8a3a813: Status 404 returned error can't find the container with id 853c953b7a7635e2785996c2dcd0269c5db56c25e172c4854e26356ac8a3a813 Oct 31 00:48:19.833336 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:19.833308 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:48:19.833395 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:19.833361 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:48:20.616585 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:20.616553 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-bdcsb" event={"ID":"45d15e31-0f2f-4c64-b35b-c133895ccea5","Type":"ContainerStarted","Data":"feda695cdca84991f8a504904889ec0af2575ae56b711db09de3fd59bec493da"} Oct 31 00:48:20.617023 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:20.616704 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-bdcsb" Oct 31 00:48:20.618180 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:20.618161 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"893e420c-f593-4011-83af-12f5eeb9466a","Type":"ContainerStarted","Data":"543b62cfc81797d64b521909cf27b4d5add75d90a82d8d112bc0b64c3b7919f4"} Oct 31 00:48:20.618265 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:20.618195 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"893e420c-f593-4011-83af-12f5eeb9466a","Type":"ContainerStarted","Data":"853c953b7a7635e2785996c2dcd0269c5db56c25e172c4854e26356ac8a3a813"} Oct 31 00:48:20.637761 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:20.637719 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-bdcsb" podStartSLOduration=2.637707002 podStartE2EDuration="2.637707002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:20.636305728 +0000 UTC m=+2937.273124468" watchObservedRunningTime="2026-10-31 00:48:20.637707002 +0000 UTC m=+2937.274525717" Oct 31 00:48:20.660030 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:20.659992 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.659981397 podStartE2EDuration="2.659981397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:20.658635267 +0000 UTC m=+2937.295453984" watchObservedRunningTime="2026-10-31 00:48:20.659981397 +0000 UTC m=+2937.296800110" Oct 31 00:48:20.897815 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:48:20.897735 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:48:25.142326 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.142293 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:48:25.146199 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.146176 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:48:25.149944 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.149913 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 31 00:48:25.149944 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.149937 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:48:25.150121 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.149989 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-sc6dq\"" Oct 31 00:48:25.150121 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.150007 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 31 00:48:25.150121 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.150039 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 31 00:48:25.161172 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.161145 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:48:25.187523 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.187500 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/85c3b81f-0dad-4b61-ac71-0d1c6737affb-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"85c3b81f-0dad-4b61-ac71-0d1c6737affb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:48:25.187632 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.187538 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/85c3b81f-0dad-4b61-ac71-0d1c6737affb-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"85c3b81f-0dad-4b61-ac71-0d1c6737affb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:48:25.187632 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.187604 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmktw\" (UniqueName: \"kubernetes.io/projected/85c3b81f-0dad-4b61-ac71-0d1c6737affb-kube-api-access-xmktw\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"85c3b81f-0dad-4b61-ac71-0d1c6737affb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:48:25.187717 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.187632 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/85c3b81f-0dad-4b61-ac71-0d1c6737affb-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"85c3b81f-0dad-4b61-ac71-0d1c6737affb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:48:25.187717 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.187650 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/85c3b81f-0dad-4b61-ac71-0d1c6737affb-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"85c3b81f-0dad-4b61-ac71-0d1c6737affb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:48:25.187717 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.187677 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/85c3b81f-0dad-4b61-ac71-0d1c6737affb-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"85c3b81f-0dad-4b61-ac71-0d1c6737affb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:48:25.187816 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.187730 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/85c3b81f-0dad-4b61-ac71-0d1c6737affb-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"85c3b81f-0dad-4b61-ac71-0d1c6737affb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:48:25.288734 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.288710 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/85c3b81f-0dad-4b61-ac71-0d1c6737affb-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"85c3b81f-0dad-4b61-ac71-0d1c6737affb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:48:25.288861 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.288746 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/85c3b81f-0dad-4b61-ac71-0d1c6737affb-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"85c3b81f-0dad-4b61-ac71-0d1c6737affb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:48:25.288861 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.288781 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/85c3b81f-0dad-4b61-ac71-0d1c6737affb-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"85c3b81f-0dad-4b61-ac71-0d1c6737affb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:48:25.288861 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.288811 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/85c3b81f-0dad-4b61-ac71-0d1c6737affb-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"85c3b81f-0dad-4b61-ac71-0d1c6737affb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:48:25.288861 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.288842 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmktw\" (UniqueName: \"kubernetes.io/projected/85c3b81f-0dad-4b61-ac71-0d1c6737affb-kube-api-access-xmktw\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"85c3b81f-0dad-4b61-ac71-0d1c6737affb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:48:25.289083 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.288867 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/85c3b81f-0dad-4b61-ac71-0d1c6737affb-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"85c3b81f-0dad-4b61-ac71-0d1c6737affb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:48:25.289083 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.289013 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/85c3b81f-0dad-4b61-ac71-0d1c6737affb-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"85c3b81f-0dad-4b61-ac71-0d1c6737affb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:48:25.289791 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.289767 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/85c3b81f-0dad-4b61-ac71-0d1c6737affb-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"85c3b81f-0dad-4b61-ac71-0d1c6737affb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:48:25.291917 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.291310 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/85c3b81f-0dad-4b61-ac71-0d1c6737affb-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"85c3b81f-0dad-4b61-ac71-0d1c6737affb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:48:25.291917 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.291815 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/85c3b81f-0dad-4b61-ac71-0d1c6737affb-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"85c3b81f-0dad-4b61-ac71-0d1c6737affb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:48:25.291917 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.291850 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/85c3b81f-0dad-4b61-ac71-0d1c6737affb-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"85c3b81f-0dad-4b61-ac71-0d1c6737affb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:48:25.292910 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.292889 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/85c3b81f-0dad-4b61-ac71-0d1c6737affb-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"85c3b81f-0dad-4b61-ac71-0d1c6737affb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:48:25.295392 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.295369 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/85c3b81f-0dad-4b61-ac71-0d1c6737affb-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"85c3b81f-0dad-4b61-ac71-0d1c6737affb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:48:25.303672 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.303646 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmktw\" (UniqueName: \"kubernetes.io/projected/85c3b81f-0dad-4b61-ac71-0d1c6737affb-kube-api-access-xmktw\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"85c3b81f-0dad-4b61-ac71-0d1c6737affb\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:48:25.372873 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.372843 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:48:25.377036 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.377017 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:25.394276 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.394133 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:48:25.455893 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.455867 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:48:25.456035 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.455976 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 31 00:48:25.456035 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.455990 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 31 00:48:25.456035 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.455994 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 31 00:48:25.456035 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.456016 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:48:25.456207 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.456077 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-k9mwh\"" Oct 31 00:48:25.456271 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.456254 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 31 00:48:25.456314 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.456294 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 31 00:48:25.456409 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.456395 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 31 00:48:25.465218 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.465200 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:48:25.490523 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.490497 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:25.490611 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.490529 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:25.490611 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.490547 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f818e247-1451-4035-b82d-a56f64d900b2-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:25.490611 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.490574 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f818e247-1451-4035-b82d-a56f64d900b2-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:25.490809 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.490788 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:25.490936 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.490816 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctfls\" (UniqueName: \"kubernetes.io/projected/f818e247-1451-4035-b82d-a56f64d900b2-kube-api-access-ctfls\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:25.490936 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.490842 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f818e247-1451-4035-b82d-a56f64d900b2-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:25.490936 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.490883 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:25.491159 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.490964 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:25.491159 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.491025 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:25.491159 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.491058 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f818e247-1451-4035-b82d-a56f64d900b2-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:25.491159 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.491104 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f818e247-1451-4035-b82d-a56f64d900b2-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:25.592526 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.592492 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:25.592658 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.592532 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctfls\" (UniqueName: \"kubernetes.io/projected/f818e247-1451-4035-b82d-a56f64d900b2-kube-api-access-ctfls\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:25.592658 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.592553 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f818e247-1451-4035-b82d-a56f64d900b2-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:25.592658 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.592569 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:25.592658 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.592584 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:25.592658 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.592611 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:25.592658 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.592639 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f818e247-1451-4035-b82d-a56f64d900b2-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:25.593011 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.592787 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f818e247-1451-4035-b82d-a56f64d900b2-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:25.593011 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.592848 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:25.593011 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.592882 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:25.593011 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.592905 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f818e247-1451-4035-b82d-a56f64d900b2-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:25.593011 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.592972 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f818e247-1451-4035-b82d-a56f64d900b2-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:25.593655 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.593587 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f818e247-1451-4035-b82d-a56f64d900b2-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:25.593771 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.593681 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f818e247-1451-4035-b82d-a56f64d900b2-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:25.594037 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.593991 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f818e247-1451-4035-b82d-a56f64d900b2-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:25.595526 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.595499 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f818e247-1451-4035-b82d-a56f64d900b2-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:25.596083 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.596059 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:25.596187 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.596099 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:25.596187 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.596152 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:25.596356 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.596336 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:25.596468 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.596445 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:25.596504 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.595944 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:25.597557 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.597540 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f818e247-1451-4035-b82d-a56f64d900b2-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:25.605682 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.605664 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctfls\" (UniqueName: \"kubernetes.io/projected/f818e247-1451-4035-b82d-a56f64d900b2-kube-api-access-ctfls\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:25.744112 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:25.744087 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:48:25.745787 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:48:25.745762 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85c3b81f_0dad_4b61_ac71_0d1c6737affb.slice/crio-242ef08e5c564b92297e9247718454c9f2b30bc056ed4fc44d24de9daa25fe85 WatchSource:0}: Error finding container 242ef08e5c564b92297e9247718454c9f2b30bc056ed4fc44d24de9daa25fe85: Status 404 returned error can't find the container with id 242ef08e5c564b92297e9247718454c9f2b30bc056ed4fc44d24de9daa25fe85 Oct 31 00:48:26.625694 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:26.625668 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-bdcsb" Oct 31 00:48:26.652929 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:26.652822 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"85c3b81f-0dad-4b61-ac71-0d1c6737affb","Type":"ContainerStarted","Data":"ec50354b3c9281e211357ff125cdb4e7e5be4b33453ff15e97a3cd9fe29a665b"} Oct 31 00:48:26.653088 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:26.652993 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"85c3b81f-0dad-4b61-ac71-0d1c6737affb","Type":"ContainerStarted","Data":"242ef08e5c564b92297e9247718454c9f2b30bc056ed4fc44d24de9daa25fe85"} Oct 31 00:48:29.668143 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:29.668114 5274 generic.go:356] "Generic (PLEG): container finished" podID="85c3b81f-0dad-4b61-ac71-0d1c6737affb" containerID="ec50354b3c9281e211357ff125cdb4e7e5be4b33453ff15e97a3cd9fe29a665b" exitCode=0 Oct 31 00:48:29.668474 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:29.668187 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"85c3b81f-0dad-4b61-ac71-0d1c6737affb","Type":"ContainerDied","Data":"ec50354b3c9281e211357ff125cdb4e7e5be4b33453ff15e97a3cd9fe29a665b"} Oct 31 00:48:30.620112 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:30.620088 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c8mg8" Oct 31 00:48:30.674936 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:30.674907 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"85c3b81f-0dad-4b61-ac71-0d1c6737affb","Type":"ContainerStarted","Data":"fd069024b262e52e595628051b259cd3d6ea2529b88b18fd5f597f1adc43b72a"} Oct 31 00:48:30.675446 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:30.674982 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"85c3b81f-0dad-4b61-ac71-0d1c6737affb","Type":"ContainerStarted","Data":"17cd760deb2cd9d0468e9513e819a9503e407cdfc2093370f22cb5283541fe4d"} Oct 31 00:48:30.675446 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:30.675155 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:48:30.702460 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:30.702417 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=2.51719932 podStartE2EDuration="5.70240517s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.18520585s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:30.700234476 +0000 UTC m=+2947.337053193" watchObservedRunningTime="2026-10-31 00:48:30.70240517 +0000 UTC m=+2947.339223885" Oct 31 00:48:32.849905 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:32.849877 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:32.856990 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:32.856949 5274 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:32.950907 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:32.950879 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:32.951038 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:32.951017 5274 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:32.954544 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:32.954527 5274 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") DevicePath \"csi-3beb91b3ebf177571ca474589da3560cd18787cfa50b2a0b9bfef85661e6cf21\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:33.046309 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:33.046285 5274 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/534eb232558c8505fa55558aa87cc96e2d2b0adb946edb717939ec387be54e97/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:33.062637 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:33.062611 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:33.197526 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:33.197503 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:33.484601 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:33.484575 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:48:33.486520 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:48:33.486494 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf818e247_1451_4035_b82d_a56f64d900b2.slice/crio-f4d00b0b1467af0e2b9f1be72cce5485c0ff1d698ac9cc7e520271b198e67ca1 WatchSource:0}: Error finding container f4d00b0b1467af0e2b9f1be72cce5485c0ff1d698ac9cc7e520271b198e67ca1: Status 404 returned error can't find the container with id f4d00b0b1467af0e2b9f1be72cce5485c0ff1d698ac9cc7e520271b198e67ca1 Oct 31 00:48:33.690148 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:33.690124 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"f818e247-1451-4035-b82d-a56f64d900b2","Type":"ContainerStarted","Data":"f4d00b0b1467af0e2b9f1be72cce5485c0ff1d698ac9cc7e520271b198e67ca1"} Oct 31 00:48:34.696674 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:34.696640 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"f818e247-1451-4035-b82d-a56f64d900b2","Type":"ContainerStarted","Data":"b7d36bebe6a182bb03a78f37bf0deca51ad071f0f80b6086a8b3ca212582ba69"} Oct 31 00:48:36.680410 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:36.680381 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:48:37.713862 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:37.713831 5274 generic.go:356] "Generic (PLEG): container finished" podID="f818e247-1451-4035-b82d-a56f64d900b2" containerID="b7d36bebe6a182bb03a78f37bf0deca51ad071f0f80b6086a8b3ca212582ba69" exitCode=0 Oct 31 00:48:37.714286 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:37.713876 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"f818e247-1451-4035-b82d-a56f64d900b2","Type":"ContainerDied","Data":"b7d36bebe6a182bb03a78f37bf0deca51ad071f0f80b6086a8b3ca212582ba69"} Oct 31 00:48:38.721858 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:38.721823 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"f818e247-1451-4035-b82d-a56f64d900b2","Type":"ContainerStarted","Data":"6183275490d9f2c3e2fbc5581fc370470dcec8eac565c4d1fdbe1e3709de0783"} Oct 31 00:48:38.722375 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:38.721887 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"f818e247-1451-4035-b82d-a56f64d900b2","Type":"ContainerStarted","Data":"74d756dabc223d56aa67f39a0b36f052ba7dcb10ed83f79a39d086cce64fef80"} Oct 31 00:48:38.722375 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:38.721912 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"f818e247-1451-4035-b82d-a56f64d900b2","Type":"ContainerStarted","Data":"83789b32419d6a21fa0ed72599a40e3c2a43fa50153c1b1e4e8855a4a89d8a30"} Oct 31 00:48:38.756905 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:38.756861 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=10.470808607 podStartE2EDuration="13.75684944s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.286040833s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:38.754049355 +0000 UTC m=+2955.390868073" watchObservedRunningTime="2026-10-31 00:48:38.75684944 +0000 UTC m=+2955.393668154" Oct 31 00:48:43.198131 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:43.198091 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:48.198372 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:48.198339 5274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:48.203658 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:48.203629 5274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:48.386892 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:48.386859 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:48:48.387110 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:48.387073 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="893e420c-f593-4011-83af-12f5eeb9466a" containerName="otc-container" containerID="cri-o://543b62cfc81797d64b521909cf27b4d5add75d90a82d8d112bc0b64c3b7919f4" gracePeriod=30 Oct 31 00:48:48.776647 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:48.776622 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:48:49.639393 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:49.639372 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:49.778312 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:49.778227 5274 generic.go:356] "Generic (PLEG): container finished" podID="893e420c-f593-4011-83af-12f5eeb9466a" containerID="543b62cfc81797d64b521909cf27b4d5add75d90a82d8d112bc0b64c3b7919f4" exitCode=0 Oct 31 00:48:49.778312 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:49.778294 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:49.778550 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:49.778317 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"893e420c-f593-4011-83af-12f5eeb9466a","Type":"ContainerDied","Data":"543b62cfc81797d64b521909cf27b4d5add75d90a82d8d112bc0b64c3b7919f4"} Oct 31 00:48:49.778550 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:49.778385 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"893e420c-f593-4011-83af-12f5eeb9466a","Type":"ContainerDied","Data":"853c953b7a7635e2785996c2dcd0269c5db56c25e172c4854e26356ac8a3a813"} Oct 31 00:48:49.778550 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:49.778415 5274 scope.go:122] "RemoveContainer" containerID="543b62cfc81797d64b521909cf27b4d5add75d90a82d8d112bc0b64c3b7919f4" Oct 31 00:48:49.787183 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:49.787168 5274 scope.go:122] "RemoveContainer" containerID="543b62cfc81797d64b521909cf27b4d5add75d90a82d8d112bc0b64c3b7919f4" Oct 31 00:48:49.787478 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:48:49.787462 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"543b62cfc81797d64b521909cf27b4d5add75d90a82d8d112bc0b64c3b7919f4\": container with ID starting with 543b62cfc81797d64b521909cf27b4d5add75d90a82d8d112bc0b64c3b7919f4 not found: ID does not exist" containerID="543b62cfc81797d64b521909cf27b4d5add75d90a82d8d112bc0b64c3b7919f4" Oct 31 00:48:49.787528 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:49.787486 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"543b62cfc81797d64b521909cf27b4d5add75d90a82d8d112bc0b64c3b7919f4"} err="failed to get container status \"543b62cfc81797d64b521909cf27b4d5add75d90a82d8d112bc0b64c3b7919f4\": rpc error: code = NotFound desc = could not find container \"543b62cfc81797d64b521909cf27b4d5add75d90a82d8d112bc0b64c3b7919f4\": container with ID starting with 543b62cfc81797d64b521909cf27b4d5add75d90a82d8d112bc0b64c3b7919f4 not found: ID does not exist" Oct 31 00:48:49.789674 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:49.789657 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/893e420c-f593-4011-83af-12f5eeb9466a-tls-certs\" (UniqueName: \"kubernetes.io/secret/893e420c-f593-4011-83af-12f5eeb9466a-tls-certs\") pod \"893e420c-f593-4011-83af-12f5eeb9466a\" (UID: \"893e420c-f593-4011-83af-12f5eeb9466a\") " Oct 31 00:48:49.789737 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:49.789713 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/893e420c-f593-4011-83af-12f5eeb9466a-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/893e420c-f593-4011-83af-12f5eeb9466a-otlp-tls-certs\") pod \"893e420c-f593-4011-83af-12f5eeb9466a\" (UID: \"893e420c-f593-4011-83af-12f5eeb9466a\") " Oct 31 00:48:49.789794 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:49.789736 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/893e420c-f593-4011-83af-12f5eeb9466a-otc-internal\" (UniqueName: \"kubernetes.io/configmap/893e420c-f593-4011-83af-12f5eeb9466a-otc-internal\") pod \"893e420c-f593-4011-83af-12f5eeb9466a\" (UID: \"893e420c-f593-4011-83af-12f5eeb9466a\") " Oct 31 00:48:49.789794 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:49.789756 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/893e420c-f593-4011-83af-12f5eeb9466a-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/893e420c-f593-4011-83af-12f5eeb9466a-data-science-collector-ta-client-cert\") pod \"893e420c-f593-4011-83af-12f5eeb9466a\" (UID: \"893e420c-f593-4011-83af-12f5eeb9466a\") " Oct 31 00:48:49.789794 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:49.789783 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/893e420c-f593-4011-83af-12f5eeb9466a-kube-api-access-zzhcd\" (UniqueName: \"kubernetes.io/projected/893e420c-f593-4011-83af-12f5eeb9466a-kube-api-access-zzhcd\") pod \"893e420c-f593-4011-83af-12f5eeb9466a\" (UID: \"893e420c-f593-4011-83af-12f5eeb9466a\") " Oct 31 00:48:49.789967 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:49.789844 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/893e420c-f593-4011-83af-12f5eeb9466a-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/893e420c-f593-4011-83af-12f5eeb9466a-monitoring-tls-certs\") pod \"893e420c-f593-4011-83af-12f5eeb9466a\" (UID: \"893e420c-f593-4011-83af-12f5eeb9466a\") " Oct 31 00:48:49.790125 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:49.790093 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/893e420c-f593-4011-83af-12f5eeb9466a-otc-internal" pod "893e420c-f593-4011-83af-12f5eeb9466a" (UID: "893e420c-f593-4011-83af-12f5eeb9466a"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:49.791943 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:49.791914 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/893e420c-f593-4011-83af-12f5eeb9466a-otlp-tls-certs" pod "893e420c-f593-4011-83af-12f5eeb9466a" (UID: "893e420c-f593-4011-83af-12f5eeb9466a"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:49.791943 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:49.791933 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/893e420c-f593-4011-83af-12f5eeb9466a-data-science-collector-ta-client-cert" pod "893e420c-f593-4011-83af-12f5eeb9466a" (UID: "893e420c-f593-4011-83af-12f5eeb9466a"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:49.792095 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:49.791974 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/893e420c-f593-4011-83af-12f5eeb9466a-kube-api-access-zzhcd" pod "893e420c-f593-4011-83af-12f5eeb9466a" (UID: "893e420c-f593-4011-83af-12f5eeb9466a"). InnerVolumeSpecName "kube-api-access-zzhcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:49.792215 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:49.792192 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/893e420c-f593-4011-83af-12f5eeb9466a-monitoring-tls-certs" pod "893e420c-f593-4011-83af-12f5eeb9466a" (UID: "893e420c-f593-4011-83af-12f5eeb9466a"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:49.792215 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:49.792202 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/893e420c-f593-4011-83af-12f5eeb9466a-tls-certs" pod "893e420c-f593-4011-83af-12f5eeb9466a" (UID: "893e420c-f593-4011-83af-12f5eeb9466a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:49.891246 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:49.891158 5274 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/893e420c-f593-4011-83af-12f5eeb9466a-monitoring-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:49.891246 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:49.891186 5274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/893e420c-f593-4011-83af-12f5eeb9466a-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:49.891246 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:49.891204 5274 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/893e420c-f593-4011-83af-12f5eeb9466a-otlp-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:49.891427 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:49.891318 5274 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/893e420c-f593-4011-83af-12f5eeb9466a-otc-internal\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:49.891427 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:49.891343 5274 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/893e420c-f593-4011-83af-12f5eeb9466a-data-science-collector-ta-client-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:49.891427 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:49.891362 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzhcd\" (UniqueName: \"kubernetes.io/projected/893e420c-f593-4011-83af-12f5eeb9466a-kube-api-access-zzhcd\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:50.100968 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:50.100928 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:48:50.107650 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:50.107628 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:48:50.136279 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:50.136257 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:48:50.136599 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:50.136587 5274 cpu_manager.go:414] "containerMap: removing container" podUID="893e420c-f593-4011-83af-12f5eeb9466a" containerName="otc-container" Oct 31 00:48:50.136642 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:50.136600 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="893e420c-f593-4011-83af-12f5eeb9466a" containerName="otc-container" Oct 31 00:48:50.136700 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:50.136692 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="893e420c-f593-4011-83af-12f5eeb9466a" containerName="otc-container" Oct 31 00:48:50.141243 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:50.141225 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:50.148193 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:50.148176 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:48:50.150306 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:50.150280 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-zfncz\"" Oct 31 00:48:50.150472 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:50.150451 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:48:50.150750 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:50.150723 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:48:50.151191 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:50.151170 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:48:50.152497 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:50.152480 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:48:50.248615 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:50.248587 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:48:50.294930 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:50.294901 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/86036a57-6788-403f-8848-eb0025209623-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"86036a57-6788-403f-8848-eb0025209623\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:50.295063 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:50.294980 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/86036a57-6788-403f-8848-eb0025209623-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"86036a57-6788-403f-8848-eb0025209623\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:50.295063 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:50.295029 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/86036a57-6788-403f-8848-eb0025209623-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"86036a57-6788-403f-8848-eb0025209623\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:50.295157 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:50.295063 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m97xr\" (UniqueName: \"kubernetes.io/projected/86036a57-6788-403f-8848-eb0025209623-kube-api-access-m97xr\") pod \"data-science-collector-collector-0\" (UID: \"86036a57-6788-403f-8848-eb0025209623\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:50.295157 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:50.295135 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/86036a57-6788-403f-8848-eb0025209623-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"86036a57-6788-403f-8848-eb0025209623\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:50.295157 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:50.295153 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86036a57-6788-403f-8848-eb0025209623-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"86036a57-6788-403f-8848-eb0025209623\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:50.395603 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:50.395577 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/86036a57-6788-403f-8848-eb0025209623-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"86036a57-6788-403f-8848-eb0025209623\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:50.395739 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:50.395617 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/86036a57-6788-403f-8848-eb0025209623-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"86036a57-6788-403f-8848-eb0025209623\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:50.395739 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:50.395634 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/86036a57-6788-403f-8848-eb0025209623-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"86036a57-6788-403f-8848-eb0025209623\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:50.395739 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:50.395662 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m97xr\" (UniqueName: \"kubernetes.io/projected/86036a57-6788-403f-8848-eb0025209623-kube-api-access-m97xr\") pod \"data-science-collector-collector-0\" (UID: \"86036a57-6788-403f-8848-eb0025209623\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:50.395739 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:50.395713 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/86036a57-6788-403f-8848-eb0025209623-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"86036a57-6788-403f-8848-eb0025209623\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:50.395739 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:50.395735 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86036a57-6788-403f-8848-eb0025209623-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"86036a57-6788-403f-8848-eb0025209623\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:50.396433 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:50.396408 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/86036a57-6788-403f-8848-eb0025209623-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"86036a57-6788-403f-8848-eb0025209623\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:50.398191 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:50.398172 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/86036a57-6788-403f-8848-eb0025209623-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"86036a57-6788-403f-8848-eb0025209623\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:50.398272 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:50.398186 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/86036a57-6788-403f-8848-eb0025209623-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"86036a57-6788-403f-8848-eb0025209623\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:50.398272 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:50.398190 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/86036a57-6788-403f-8848-eb0025209623-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"86036a57-6788-403f-8848-eb0025209623\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:50.398344 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:50.398272 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86036a57-6788-403f-8848-eb0025209623-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"86036a57-6788-403f-8848-eb0025209623\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:50.408384 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:50.408362 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m97xr\" (UniqueName: \"kubernetes.io/projected/86036a57-6788-403f-8848-eb0025209623-kube-api-access-m97xr\") pod \"data-science-collector-collector-0\" (UID: \"86036a57-6788-403f-8848-eb0025209623\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:50.455472 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:50.455455 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:48:50.733093 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:50.733069 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:48:50.735109 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:48:50.735083 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86036a57_6788_403f_8848_eb0025209623.slice/crio-7166797a536d28975ff1dcf6ed923f1f3faebc96a3a6cb70a594c3c60ef66b36 WatchSource:0}: Error finding container 7166797a536d28975ff1dcf6ed923f1f3faebc96a3a6cb70a594c3c60ef66b36: Status 404 returned error can't find the container with id 7166797a536d28975ff1dcf6ed923f1f3faebc96a3a6cb70a594c3c60ef66b36 Oct 31 00:48:50.737369 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:50.737332 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:48:50.737435 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:50.737409 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:48:50.784923 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:50.784895 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"86036a57-6788-403f-8848-eb0025209623","Type":"ContainerStarted","Data":"7166797a536d28975ff1dcf6ed923f1f3faebc96a3a6cb70a594c3c60ef66b36"} Oct 31 00:48:50.898950 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:48:50.898908 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:48:51.481651 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:51.481616 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="893e420c-f593-4011-83af-12f5eeb9466a" path="/var/lib/kubelet/pods/893e420c-f593-4011-83af-12f5eeb9466a/volumes" Oct 31 00:48:51.791989 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:51.791895 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"86036a57-6788-403f-8848-eb0025209623","Type":"ContainerStarted","Data":"96873344af33a612572f66f027a63057e0f04469277890a362cce7640b685229"} Oct 31 00:48:51.814485 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:48:51.814426 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.8144139620000002 podStartE2EDuration="1.814413962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:51.812304594 +0000 UTC m=+2968.449123310" watchObservedRunningTime="2026-10-31 00:48:51.814413962 +0000 UTC m=+2968.451232677" Oct 31 00:49:01.454965 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:01.454917 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:49:01.455540 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:01.455126 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="86036a57-6788-403f-8848-eb0025209623" containerName="otc-container" containerID="cri-o://96873344af33a612572f66f027a63057e0f04469277890a362cce7640b685229" gracePeriod=30 Oct 31 00:49:02.698597 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.698574 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:02.795927 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.795855 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86036a57-6788-403f-8848-eb0025209623-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/86036a57-6788-403f-8848-eb0025209623-data-science-collector-ta-client-cert\") pod \"86036a57-6788-403f-8848-eb0025209623\" (UID: \"86036a57-6788-403f-8848-eb0025209623\") " Oct 31 00:49:02.795927 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.795921 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86036a57-6788-403f-8848-eb0025209623-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/86036a57-6788-403f-8848-eb0025209623-monitoring-tls-certs\") pod \"86036a57-6788-403f-8848-eb0025209623\" (UID: \"86036a57-6788-403f-8848-eb0025209623\") " Oct 31 00:49:02.796120 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.795981 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86036a57-6788-403f-8848-eb0025209623-tls-certs\" (UniqueName: \"kubernetes.io/secret/86036a57-6788-403f-8848-eb0025209623-tls-certs\") pod \"86036a57-6788-403f-8848-eb0025209623\" (UID: \"86036a57-6788-403f-8848-eb0025209623\") " Oct 31 00:49:02.796120 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.796025 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86036a57-6788-403f-8848-eb0025209623-kube-api-access-m97xr\" (UniqueName: \"kubernetes.io/projected/86036a57-6788-403f-8848-eb0025209623-kube-api-access-m97xr\") pod \"86036a57-6788-403f-8848-eb0025209623\" (UID: \"86036a57-6788-403f-8848-eb0025209623\") " Oct 31 00:49:02.796120 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.796096 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86036a57-6788-403f-8848-eb0025209623-otc-internal\" (UniqueName: \"kubernetes.io/configmap/86036a57-6788-403f-8848-eb0025209623-otc-internal\") pod \"86036a57-6788-403f-8848-eb0025209623\" (UID: \"86036a57-6788-403f-8848-eb0025209623\") " Oct 31 00:49:02.796240 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.796215 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86036a57-6788-403f-8848-eb0025209623-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/86036a57-6788-403f-8848-eb0025209623-otlp-tls-certs\") pod \"86036a57-6788-403f-8848-eb0025209623\" (UID: \"86036a57-6788-403f-8848-eb0025209623\") " Oct 31 00:49:02.796470 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.796442 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86036a57-6788-403f-8848-eb0025209623-otc-internal" pod "86036a57-6788-403f-8848-eb0025209623" (UID: "86036a57-6788-403f-8848-eb0025209623"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:02.796601 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.796550 5274 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/86036a57-6788-403f-8848-eb0025209623-otc-internal\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:02.798239 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.798217 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86036a57-6788-403f-8848-eb0025209623-tls-certs" pod "86036a57-6788-403f-8848-eb0025209623" (UID: "86036a57-6788-403f-8848-eb0025209623"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:02.798537 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.798514 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86036a57-6788-403f-8848-eb0025209623-monitoring-tls-certs" pod "86036a57-6788-403f-8848-eb0025209623" (UID: "86036a57-6788-403f-8848-eb0025209623"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:02.798721 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.798694 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86036a57-6788-403f-8848-eb0025209623-kube-api-access-m97xr" pod "86036a57-6788-403f-8848-eb0025209623" (UID: "86036a57-6788-403f-8848-eb0025209623"). InnerVolumeSpecName "kube-api-access-m97xr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:02.798808 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.798768 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86036a57-6788-403f-8848-eb0025209623-data-science-collector-ta-client-cert" pod "86036a57-6788-403f-8848-eb0025209623" (UID: "86036a57-6788-403f-8848-eb0025209623"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:02.798908 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.798887 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86036a57-6788-403f-8848-eb0025209623-otlp-tls-certs" pod "86036a57-6788-403f-8848-eb0025209623" (UID: "86036a57-6788-403f-8848-eb0025209623"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:02.847941 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.847916 5274 generic.go:356] "Generic (PLEG): container finished" podID="86036a57-6788-403f-8848-eb0025209623" containerID="96873344af33a612572f66f027a63057e0f04469277890a362cce7640b685229" exitCode=0 Oct 31 00:49:02.848050 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.847982 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:02.848050 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.847997 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"86036a57-6788-403f-8848-eb0025209623","Type":"ContainerDied","Data":"96873344af33a612572f66f027a63057e0f04469277890a362cce7640b685229"} Oct 31 00:49:02.848050 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.848040 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"86036a57-6788-403f-8848-eb0025209623","Type":"ContainerDied","Data":"7166797a536d28975ff1dcf6ed923f1f3faebc96a3a6cb70a594c3c60ef66b36"} Oct 31 00:49:02.848176 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.848060 5274 scope.go:122] "RemoveContainer" containerID="96873344af33a612572f66f027a63057e0f04469277890a362cce7640b685229" Oct 31 00:49:02.857277 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.857263 5274 scope.go:122] "RemoveContainer" containerID="96873344af33a612572f66f027a63057e0f04469277890a362cce7640b685229" Oct 31 00:49:02.857545 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:49:02.857526 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96873344af33a612572f66f027a63057e0f04469277890a362cce7640b685229\": container with ID starting with 96873344af33a612572f66f027a63057e0f04469277890a362cce7640b685229 not found: ID does not exist" containerID="96873344af33a612572f66f027a63057e0f04469277890a362cce7640b685229" Oct 31 00:49:02.857592 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.857553 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96873344af33a612572f66f027a63057e0f04469277890a362cce7640b685229"} err="failed to get container status \"96873344af33a612572f66f027a63057e0f04469277890a362cce7640b685229\": rpc error: code = NotFound desc = could not find container \"96873344af33a612572f66f027a63057e0f04469277890a362cce7640b685229\": container with ID starting with 96873344af33a612572f66f027a63057e0f04469277890a362cce7640b685229 not found: ID does not exist" Oct 31 00:49:02.870596 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.870574 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:49:02.874880 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.874860 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:49:02.896988 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.896945 5274 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/86036a57-6788-403f-8848-eb0025209623-data-science-collector-ta-client-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:02.896988 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.896986 5274 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/86036a57-6788-403f-8848-eb0025209623-monitoring-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:02.897111 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.896995 5274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86036a57-6788-403f-8848-eb0025209623-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:02.897111 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.897006 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m97xr\" (UniqueName: \"kubernetes.io/projected/86036a57-6788-403f-8848-eb0025209623-kube-api-access-m97xr\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:02.897111 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.897017 5274 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/86036a57-6788-403f-8848-eb0025209623-otlp-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:02.902193 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.902172 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:49:02.902532 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.902520 5274 cpu_manager.go:414] "containerMap: removing container" podUID="86036a57-6788-403f-8848-eb0025209623" containerName="otc-container" Oct 31 00:49:02.902578 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.902534 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86036a57-6788-403f-8848-eb0025209623" containerName="otc-container" Oct 31 00:49:02.902624 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.902615 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="86036a57-6788-403f-8848-eb0025209623" containerName="otc-container" Oct 31 00:49:02.912941 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.912920 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:02.917239 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.917205 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:49:02.918673 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.918072 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:49:02.918673 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.918147 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-zfncz\"" Oct 31 00:49:02.918673 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.918153 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:49:02.918673 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.918306 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:49:02.918673 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.918352 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:49:02.919287 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:02.919253 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:49:03.098692 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:03.098619 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5226ae19-6e03-49a3-92f5-8d519cd787ac-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5226ae19-6e03-49a3-92f5-8d519cd787ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:03.098692 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:03.098676 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5226ae19-6e03-49a3-92f5-8d519cd787ac-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5226ae19-6e03-49a3-92f5-8d519cd787ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:03.098886 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:03.098749 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5226ae19-6e03-49a3-92f5-8d519cd787ac-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5226ae19-6e03-49a3-92f5-8d519cd787ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:03.098886 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:03.098793 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxnpt\" (UniqueName: \"kubernetes.io/projected/5226ae19-6e03-49a3-92f5-8d519cd787ac-kube-api-access-vxnpt\") pod \"data-science-collector-collector-0\" (UID: \"5226ae19-6e03-49a3-92f5-8d519cd787ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:03.098886 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:03.098834 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5226ae19-6e03-49a3-92f5-8d519cd787ac-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5226ae19-6e03-49a3-92f5-8d519cd787ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:03.099039 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:03.098920 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5226ae19-6e03-49a3-92f5-8d519cd787ac-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5226ae19-6e03-49a3-92f5-8d519cd787ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:03.199585 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:03.199561 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5226ae19-6e03-49a3-92f5-8d519cd787ac-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5226ae19-6e03-49a3-92f5-8d519cd787ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:03.199733 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:03.199605 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5226ae19-6e03-49a3-92f5-8d519cd787ac-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5226ae19-6e03-49a3-92f5-8d519cd787ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:03.199733 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:03.199658 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5226ae19-6e03-49a3-92f5-8d519cd787ac-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5226ae19-6e03-49a3-92f5-8d519cd787ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:03.199733 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:03.199709 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5226ae19-6e03-49a3-92f5-8d519cd787ac-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5226ae19-6e03-49a3-92f5-8d519cd787ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:03.199908 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:03.199749 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5226ae19-6e03-49a3-92f5-8d519cd787ac-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5226ae19-6e03-49a3-92f5-8d519cd787ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:03.199908 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:03.199775 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxnpt\" (UniqueName: \"kubernetes.io/projected/5226ae19-6e03-49a3-92f5-8d519cd787ac-kube-api-access-vxnpt\") pod \"data-science-collector-collector-0\" (UID: \"5226ae19-6e03-49a3-92f5-8d519cd787ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:03.200479 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:03.200449 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5226ae19-6e03-49a3-92f5-8d519cd787ac-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5226ae19-6e03-49a3-92f5-8d519cd787ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:03.202354 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:03.202330 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5226ae19-6e03-49a3-92f5-8d519cd787ac-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5226ae19-6e03-49a3-92f5-8d519cd787ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:03.202458 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:03.202366 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5226ae19-6e03-49a3-92f5-8d519cd787ac-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5226ae19-6e03-49a3-92f5-8d519cd787ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:03.202458 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:03.202400 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5226ae19-6e03-49a3-92f5-8d519cd787ac-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5226ae19-6e03-49a3-92f5-8d519cd787ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:03.202540 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:03.202487 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5226ae19-6e03-49a3-92f5-8d519cd787ac-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5226ae19-6e03-49a3-92f5-8d519cd787ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:03.212296 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:03.212274 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxnpt\" (UniqueName: \"kubernetes.io/projected/5226ae19-6e03-49a3-92f5-8d519cd787ac-kube-api-access-vxnpt\") pod \"data-science-collector-collector-0\" (UID: \"5226ae19-6e03-49a3-92f5-8d519cd787ac\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:03.227102 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:03.227086 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:03.482146 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:03.482120 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86036a57-6788-403f-8848-eb0025209623" path="/var/lib/kubelet/pods/86036a57-6788-403f-8848-eb0025209623/volumes" Oct 31 00:49:03.500401 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:03.500360 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:49:03.502529 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:49:03.502504 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5226ae19_6e03_49a3_92f5_8d519cd787ac.slice/crio-dc94f80334b1e37708686cd0be51a03574cc7ae50766d002af86055df2c51966 WatchSource:0}: Error finding container dc94f80334b1e37708686cd0be51a03574cc7ae50766d002af86055df2c51966: Status 404 returned error can't find the container with id dc94f80334b1e37708686cd0be51a03574cc7ae50766d002af86055df2c51966 Oct 31 00:49:03.504497 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:03.504465 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:49:03.504580 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:03.504539 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:49:03.855109 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:03.855021 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5226ae19-6e03-49a3-92f5-8d519cd787ac","Type":"ContainerStarted","Data":"769f4aed06d094eac3ded0c8199231dea1dd73ab5137e35c1503817cc8d35e98"} Oct 31 00:49:03.855109 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:03.855068 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5226ae19-6e03-49a3-92f5-8d519cd787ac","Type":"ContainerStarted","Data":"dc94f80334b1e37708686cd0be51a03574cc7ae50766d002af86055df2c51966"} Oct 31 00:49:03.876577 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:03.876487 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.876471768 podStartE2EDuration="1.876471768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:03.87578944 +0000 UTC m=+2980.512608157" watchObservedRunningTime="2026-10-31 00:49:03.876471768 +0000 UTC m=+2980.513290485" Oct 31 00:49:04.833766 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:04.833727 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-bdcsb"] Oct 31 00:49:04.834037 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:04.833990 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-bdcsb" podUID="45d15e31-0f2f-4c64-b35b-c133895ccea5" containerName="kube-rbac-proxy" containerID="cri-o://feda695cdca84991f8a504904889ec0af2575ae56b711db09de3fd59bec493da" gracePeriod=30 Oct 31 00:49:04.848623 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:04.848597 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-c8mg8"] Oct 31 00:49:04.848801 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:04.848782 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c8mg8" podUID="e662ea68-3516-4efd-8e66-f86532b11225" containerName="korrel8r" containerID="cri-o://a01c811b9b6ee96f60c8a7dac44549b08b9e66ccfa601498827d765dbbf42882" gracePeriod=30 Oct 31 00:49:05.052638 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.052611 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:49:05.053402 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.053368 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="85c3b81f-0dad-4b61-ac71-0d1c6737affb" containerName="alertmanager" containerID="cri-o://17cd760deb2cd9d0468e9513e819a9503e407cdfc2093370f22cb5283541fe4d" gracePeriod=120 Oct 31 00:49:05.053605 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.053401 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="85c3b81f-0dad-4b61-ac71-0d1c6737affb" containerName="config-reloader" containerID="cri-o://fd069024b262e52e595628051b259cd3d6ea2529b88b18fd5f597f1adc43b72a" gracePeriod=120 Oct 31 00:49:05.063990 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.063588 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:49:05.063990 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.063836 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="f818e247-1451-4035-b82d-a56f64d900b2" containerName="prometheus" containerID="cri-o://83789b32419d6a21fa0ed72599a40e3c2a43fa50153c1b1e4e8855a4a89d8a30" gracePeriod=600 Oct 31 00:49:05.063990 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.063848 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="f818e247-1451-4035-b82d-a56f64d900b2" containerName="thanos-sidecar" containerID="cri-o://6183275490d9f2c3e2fbc5581fc370470dcec8eac565c4d1fdbe1e3709de0783" gracePeriod=600 Oct 31 00:49:05.063990 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.063940 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="f818e247-1451-4035-b82d-a56f64d900b2" containerName="config-reloader" containerID="cri-o://74d756dabc223d56aa67f39a0b36f052ba7dcb10ed83f79a39d086cce64fef80" gracePeriod=600 Oct 31 00:49:05.184155 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.184119 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-bdcsb" Oct 31 00:49:05.186673 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.186282 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c8mg8" Oct 31 00:49:05.238112 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.238079 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:49:05.318441 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.318406 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45d15e31-0f2f-4c64-b35b-c133895ccea5-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/45d15e31-0f2f-4c64-b35b-c133895ccea5-prometheus-ca\") pod \"45d15e31-0f2f-4c64-b35b-c133895ccea5\" (UID: \"45d15e31-0f2f-4c64-b35b-c133895ccea5\") " Oct 31 00:49:05.318636 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.318508 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45d15e31-0f2f-4c64-b35b-c133895ccea5-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45d15e31-0f2f-4c64-b35b-c133895ccea5-kube-rbac-proxy-config\") pod \"45d15e31-0f2f-4c64-b35b-c133895ccea5\" (UID: \"45d15e31-0f2f-4c64-b35b-c133895ccea5\") " Oct 31 00:49:05.318636 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.318540 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e662ea68-3516-4efd-8e66-f86532b11225-kube-api-access-8wzs5\" (UniqueName: \"kubernetes.io/projected/e662ea68-3516-4efd-8e66-f86532b11225-kube-api-access-8wzs5\") pod \"e662ea68-3516-4efd-8e66-f86532b11225\" (UID: \"e662ea68-3516-4efd-8e66-f86532b11225\") " Oct 31 00:49:05.318636 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.318577 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e662ea68-3516-4efd-8e66-f86532b11225-config\" (UniqueName: \"kubernetes.io/configmap/e662ea68-3516-4efd-8e66-f86532b11225-config\") pod \"e662ea68-3516-4efd-8e66-f86532b11225\" (UID: \"e662ea68-3516-4efd-8e66-f86532b11225\") " Oct 31 00:49:05.318832 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.318796 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45d15e31-0f2f-4c64-b35b-c133895ccea5-prometheus-ca" pod "45d15e31-0f2f-4c64-b35b-c133895ccea5" (UID: "45d15e31-0f2f-4c64-b35b-c133895ccea5"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:05.318898 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.318838 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45d15e31-0f2f-4c64-b35b-c133895ccea5-tls\" (UniqueName: \"kubernetes.io/secret/45d15e31-0f2f-4c64-b35b-c133895ccea5-tls\") pod \"45d15e31-0f2f-4c64-b35b-c133895ccea5\" (UID: \"45d15e31-0f2f-4c64-b35b-c133895ccea5\") " Oct 31 00:49:05.318972 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.318893 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45d15e31-0f2f-4c64-b35b-c133895ccea5-kube-api-access-4jlt6\" (UniqueName: \"kubernetes.io/projected/45d15e31-0f2f-4c64-b35b-c133895ccea5-kube-api-access-4jlt6\") pod \"45d15e31-0f2f-4c64-b35b-c133895ccea5\" (UID: \"45d15e31-0f2f-4c64-b35b-c133895ccea5\") " Oct 31 00:49:05.318972 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.318934 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45d15e31-0f2f-4c64-b35b-c133895ccea5-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/45d15e31-0f2f-4c64-b35b-c133895ccea5-prometheus-client-cert\") pod \"45d15e31-0f2f-4c64-b35b-c133895ccea5\" (UID: \"45d15e31-0f2f-4c64-b35b-c133895ccea5\") " Oct 31 00:49:05.318972 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.318939 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e662ea68-3516-4efd-8e66-f86532b11225-config" pod "e662ea68-3516-4efd-8e66-f86532b11225" (UID: "e662ea68-3516-4efd-8e66-f86532b11225"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:05.319151 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.319002 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e662ea68-3516-4efd-8e66-f86532b11225-tls\" (UniqueName: \"kubernetes.io/secret/e662ea68-3516-4efd-8e66-f86532b11225-tls\") pod \"e662ea68-3516-4efd-8e66-f86532b11225\" (UID: \"e662ea68-3516-4efd-8e66-f86532b11225\") " Oct 31 00:49:05.319362 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.319337 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/45d15e31-0f2f-4c64-b35b-c133895ccea5-prometheus-ca\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.319471 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.319361 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e662ea68-3516-4efd-8e66-f86532b11225-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.349526 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.349445 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e662ea68-3516-4efd-8e66-f86532b11225-tls" pod "e662ea68-3516-4efd-8e66-f86532b11225" (UID: "e662ea68-3516-4efd-8e66-f86532b11225"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:05.349526 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.349472 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45d15e31-0f2f-4c64-b35b-c133895ccea5-kube-api-access-4jlt6" pod "45d15e31-0f2f-4c64-b35b-c133895ccea5" (UID: "45d15e31-0f2f-4c64-b35b-c133895ccea5"). InnerVolumeSpecName "kube-api-access-4jlt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:05.349526 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.349458 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e662ea68-3516-4efd-8e66-f86532b11225-kube-api-access-8wzs5" pod "e662ea68-3516-4efd-8e66-f86532b11225" (UID: "e662ea68-3516-4efd-8e66-f86532b11225"). InnerVolumeSpecName "kube-api-access-8wzs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:05.349844 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.349575 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45d15e31-0f2f-4c64-b35b-c133895ccea5-prometheus-client-cert" pod "45d15e31-0f2f-4c64-b35b-c133895ccea5" (UID: "45d15e31-0f2f-4c64-b35b-c133895ccea5"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:05.349844 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.349637 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45d15e31-0f2f-4c64-b35b-c133895ccea5-tls" pod "45d15e31-0f2f-4c64-b35b-c133895ccea5" (UID: "45d15e31-0f2f-4c64-b35b-c133895ccea5"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:05.349844 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.349678 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45d15e31-0f2f-4c64-b35b-c133895ccea5-kube-rbac-proxy-config" pod "45d15e31-0f2f-4c64-b35b-c133895ccea5" (UID: "45d15e31-0f2f-4c64-b35b-c133895ccea5"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:05.420160 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.420133 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/45d15e31-0f2f-4c64-b35b-c133895ccea5-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.420160 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.420157 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jlt6\" (UniqueName: \"kubernetes.io/projected/45d15e31-0f2f-4c64-b35b-c133895ccea5-kube-api-access-4jlt6\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.420160 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.420167 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/45d15e31-0f2f-4c64-b35b-c133895ccea5-prometheus-client-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.420456 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.420177 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e662ea68-3516-4efd-8e66-f86532b11225-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.420456 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.420185 5274 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45d15e31-0f2f-4c64-b35b-c133895ccea5-kube-rbac-proxy-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.420456 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.420193 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wzs5\" (UniqueName: \"kubernetes.io/projected/e662ea68-3516-4efd-8e66-f86532b11225-kube-api-access-8wzs5\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.868447 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.868421 5274 generic.go:356] "Generic (PLEG): container finished" podID="85c3b81f-0dad-4b61-ac71-0d1c6737affb" containerID="fd069024b262e52e595628051b259cd3d6ea2529b88b18fd5f597f1adc43b72a" exitCode=0 Oct 31 00:49:05.868569 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.868490 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"85c3b81f-0dad-4b61-ac71-0d1c6737affb","Type":"ContainerDied","Data":"fd069024b262e52e595628051b259cd3d6ea2529b88b18fd5f597f1adc43b72a"} Oct 31 00:49:05.869890 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.869865 5274 generic.go:356] "Generic (PLEG): container finished" podID="45d15e31-0f2f-4c64-b35b-c133895ccea5" containerID="feda695cdca84991f8a504904889ec0af2575ae56b711db09de3fd59bec493da" exitCode=0 Oct 31 00:49:05.870041 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.869949 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-bdcsb" Oct 31 00:49:05.870041 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.869985 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-bdcsb" event={"ID":"45d15e31-0f2f-4c64-b35b-c133895ccea5","Type":"ContainerDied","Data":"feda695cdca84991f8a504904889ec0af2575ae56b711db09de3fd59bec493da"} Oct 31 00:49:05.870149 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.870055 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-bdcsb" event={"ID":"45d15e31-0f2f-4c64-b35b-c133895ccea5","Type":"ContainerDied","Data":"ec78552903df1b29df579c8ae3630f2d15110392eddd08b4a21dc4d9683f408d"} Oct 31 00:49:05.870149 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.870084 5274 scope.go:122] "RemoveContainer" containerID="feda695cdca84991f8a504904889ec0af2575ae56b711db09de3fd59bec493da" Oct 31 00:49:05.872488 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.872464 5274 generic.go:356] "Generic (PLEG): container finished" podID="f818e247-1451-4035-b82d-a56f64d900b2" containerID="6183275490d9f2c3e2fbc5581fc370470dcec8eac565c4d1fdbe1e3709de0783" exitCode=0 Oct 31 00:49:05.872587 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.872491 5274 generic.go:356] "Generic (PLEG): container finished" podID="f818e247-1451-4035-b82d-a56f64d900b2" containerID="74d756dabc223d56aa67f39a0b36f052ba7dcb10ed83f79a39d086cce64fef80" exitCode=0 Oct 31 00:49:05.872587 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.872505 5274 generic.go:356] "Generic (PLEG): container finished" podID="f818e247-1451-4035-b82d-a56f64d900b2" containerID="83789b32419d6a21fa0ed72599a40e3c2a43fa50153c1b1e4e8855a4a89d8a30" exitCode=0 Oct 31 00:49:05.872587 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.872532 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"f818e247-1451-4035-b82d-a56f64d900b2","Type":"ContainerDied","Data":"6183275490d9f2c3e2fbc5581fc370470dcec8eac565c4d1fdbe1e3709de0783"} Oct 31 00:49:05.872750 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.872595 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"f818e247-1451-4035-b82d-a56f64d900b2","Type":"ContainerDied","Data":"74d756dabc223d56aa67f39a0b36f052ba7dcb10ed83f79a39d086cce64fef80"} Oct 31 00:49:05.872750 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.872633 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"f818e247-1451-4035-b82d-a56f64d900b2","Type":"ContainerDied","Data":"83789b32419d6a21fa0ed72599a40e3c2a43fa50153c1b1e4e8855a4a89d8a30"} Oct 31 00:49:05.872750 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.872660 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"f818e247-1451-4035-b82d-a56f64d900b2","Type":"ContainerDied","Data":"f4d00b0b1467af0e2b9f1be72cce5485c0ff1d698ac9cc7e520271b198e67ca1"} Oct 31 00:49:05.872750 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.872693 5274 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4d00b0b1467af0e2b9f1be72cce5485c0ff1d698ac9cc7e520271b198e67ca1" Oct 31 00:49:05.873880 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.873856 5274 generic.go:356] "Generic (PLEG): container finished" podID="e662ea68-3516-4efd-8e66-f86532b11225" containerID="a01c811b9b6ee96f60c8a7dac44549b08b9e66ccfa601498827d765dbbf42882" exitCode=0 Oct 31 00:49:05.874064 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.873910 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c8mg8" event={"ID":"e662ea68-3516-4efd-8e66-f86532b11225","Type":"ContainerDied","Data":"a01c811b9b6ee96f60c8a7dac44549b08b9e66ccfa601498827d765dbbf42882"} Oct 31 00:49:05.874064 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.873976 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c8mg8" event={"ID":"e662ea68-3516-4efd-8e66-f86532b11225","Type":"ContainerDied","Data":"cfa14bb9519e8eabcb78562280f733dc31fa749bedc248b7772244a50248d526"} Oct 31 00:49:05.874064 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.873918 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-c8mg8" Oct 31 00:49:05.874503 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.874478 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="5226ae19-6e03-49a3-92f5-8d519cd787ac" containerName="otc-container" containerID="cri-o://769f4aed06d094eac3ded0c8199231dea1dd73ab5137e35c1503817cc8d35e98" gracePeriod=30 Oct 31 00:49:05.930554 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.930522 5274 scope.go:122] "RemoveContainer" containerID="feda695cdca84991f8a504904889ec0af2575ae56b711db09de3fd59bec493da" Oct 31 00:49:05.931258 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:49:05.931084 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feda695cdca84991f8a504904889ec0af2575ae56b711db09de3fd59bec493da\": container with ID starting with feda695cdca84991f8a504904889ec0af2575ae56b711db09de3fd59bec493da not found: ID does not exist" containerID="feda695cdca84991f8a504904889ec0af2575ae56b711db09de3fd59bec493da" Oct 31 00:49:05.931258 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.931122 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feda695cdca84991f8a504904889ec0af2575ae56b711db09de3fd59bec493da"} err="failed to get container status \"feda695cdca84991f8a504904889ec0af2575ae56b711db09de3fd59bec493da\": rpc error: code = NotFound desc = could not find container \"feda695cdca84991f8a504904889ec0af2575ae56b711db09de3fd59bec493da\": container with ID starting with feda695cdca84991f8a504904889ec0af2575ae56b711db09de3fd59bec493da not found: ID does not exist" Oct 31 00:49:05.931258 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.931146 5274 scope.go:122] "RemoveContainer" containerID="a01c811b9b6ee96f60c8a7dac44549b08b9e66ccfa601498827d765dbbf42882" Oct 31 00:49:05.940146 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.940123 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:05.950343 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.950314 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-c8mg8"] Oct 31 00:49:05.954760 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.954737 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-c8mg8"] Oct 31 00:49:05.966445 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.966424 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-bdcsb"] Oct 31 00:49:05.970622 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.970602 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-bdcsb"] Oct 31 00:49:05.985615 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.985594 5274 scope.go:122] "RemoveContainer" containerID="a01c811b9b6ee96f60c8a7dac44549b08b9e66ccfa601498827d765dbbf42882" Oct 31 00:49:05.985998 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:49:05.985977 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a01c811b9b6ee96f60c8a7dac44549b08b9e66ccfa601498827d765dbbf42882\": container with ID starting with a01c811b9b6ee96f60c8a7dac44549b08b9e66ccfa601498827d765dbbf42882 not found: ID does not exist" containerID="a01c811b9b6ee96f60c8a7dac44549b08b9e66ccfa601498827d765dbbf42882" Oct 31 00:49:05.986117 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:05.986023 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a01c811b9b6ee96f60c8a7dac44549b08b9e66ccfa601498827d765dbbf42882"} err="failed to get container status \"a01c811b9b6ee96f60c8a7dac44549b08b9e66ccfa601498827d765dbbf42882\": rpc error: code = NotFound desc = could not find container \"a01c811b9b6ee96f60c8a7dac44549b08b9e66ccfa601498827d765dbbf42882\": container with ID starting with a01c811b9b6ee96f60c8a7dac44549b08b9e66ccfa601498827d765dbbf42882 not found: ID does not exist" Oct 31 00:49:06.008681 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.008656 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:49:06.037013 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.036985 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"f818e247-1451-4035-b82d-a56f64d900b2\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " Oct 31 00:49:06.037200 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.037036 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85c3b81f-0dad-4b61-ac71-0d1c6737affb-kube-api-access-xmktw\" (UniqueName: \"kubernetes.io/projected/85c3b81f-0dad-4b61-ac71-0d1c6737affb-kube-api-access-xmktw\") pod \"85c3b81f-0dad-4b61-ac71-0d1c6737affb\" (UID: \"85c3b81f-0dad-4b61-ac71-0d1c6737affb\") " Oct 31 00:49:06.037200 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.037076 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f818e247-1451-4035-b82d-a56f64d900b2-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f818e247-1451-4035-b82d-a56f64d900b2-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"f818e247-1451-4035-b82d-a56f64d900b2\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " Oct 31 00:49:06.037200 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.037108 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85c3b81f-0dad-4b61-ac71-0d1c6737affb-web-config\" (UniqueName: \"kubernetes.io/secret/85c3b81f-0dad-4b61-ac71-0d1c6737affb-web-config\") pod \"85c3b81f-0dad-4b61-ac71-0d1c6737affb\" (UID: \"85c3b81f-0dad-4b61-ac71-0d1c6737affb\") " Oct 31 00:49:06.037200 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.037139 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f818e247-1451-4035-b82d-a56f64d900b2-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f818e247-1451-4035-b82d-a56f64d900b2-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"f818e247-1451-4035-b82d-a56f64d900b2\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " Oct 31 00:49:06.037200 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.037180 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85c3b81f-0dad-4b61-ac71-0d1c6737affb-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/85c3b81f-0dad-4b61-ac71-0d1c6737affb-cluster-tls-config\") pod \"85c3b81f-0dad-4b61-ac71-0d1c6737affb\" (UID: \"85c3b81f-0dad-4b61-ac71-0d1c6737affb\") " Oct 31 00:49:06.037472 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.037216 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85c3b81f-0dad-4b61-ac71-0d1c6737affb-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/85c3b81f-0dad-4b61-ac71-0d1c6737affb-alertmanager-data-science-monitoringstack-db\") pod \"85c3b81f-0dad-4b61-ac71-0d1c6737affb\" (UID: \"85c3b81f-0dad-4b61-ac71-0d1c6737affb\") " Oct 31 00:49:06.037472 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.037242 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-secret-prometheus-web-tls-ca\") pod \"f818e247-1451-4035-b82d-a56f64d900b2\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " Oct 31 00:49:06.037472 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.037264 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85c3b81f-0dad-4b61-ac71-0d1c6737affb-config-volume\" (UniqueName: \"kubernetes.io/secret/85c3b81f-0dad-4b61-ac71-0d1c6737affb-config-volume\") pod \"85c3b81f-0dad-4b61-ac71-0d1c6737affb\" (UID: \"85c3b81f-0dad-4b61-ac71-0d1c6737affb\") " Oct 31 00:49:06.037472 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.037284 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f818e247-1451-4035-b82d-a56f64d900b2-tls-assets\" (UniqueName: \"kubernetes.io/projected/f818e247-1451-4035-b82d-a56f64d900b2-tls-assets\") pod \"f818e247-1451-4035-b82d-a56f64d900b2\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " Oct 31 00:49:06.037472 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.037305 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f818e247-1451-4035-b82d-a56f64d900b2-kube-api-access-ctfls\" (UniqueName: \"kubernetes.io/projected/f818e247-1451-4035-b82d-a56f64d900b2-kube-api-access-ctfls\") pod \"f818e247-1451-4035-b82d-a56f64d900b2\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " Oct 31 00:49:06.037472 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.037327 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85c3b81f-0dad-4b61-ac71-0d1c6737affb-config-out\" (UniqueName: \"kubernetes.io/empty-dir/85c3b81f-0dad-4b61-ac71-0d1c6737affb-config-out\") pod \"85c3b81f-0dad-4b61-ac71-0d1c6737affb\" (UID: \"85c3b81f-0dad-4b61-ac71-0d1c6737affb\") " Oct 31 00:49:06.037472 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.037352 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"f818e247-1451-4035-b82d-a56f64d900b2\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " Oct 31 00:49:06.037472 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.037376 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"f818e247-1451-4035-b82d-a56f64d900b2\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " Oct 31 00:49:06.037472 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.037407 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-web-config\" (UniqueName: \"kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-web-config\") pod \"f818e247-1451-4035-b82d-a56f64d900b2\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " Oct 31 00:49:06.037472 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.037433 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-config\" (UniqueName: \"kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-config\") pod \"f818e247-1451-4035-b82d-a56f64d900b2\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " Oct 31 00:49:06.037472 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.037462 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85c3b81f-0dad-4b61-ac71-0d1c6737affb-tls-assets\" (UniqueName: \"kubernetes.io/projected/85c3b81f-0dad-4b61-ac71-0d1c6737affb-tls-assets\") pod \"85c3b81f-0dad-4b61-ac71-0d1c6737affb\" (UID: \"85c3b81f-0dad-4b61-ac71-0d1c6737affb\") " Oct 31 00:49:06.038275 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.037518 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-thanos-prometheus-http-client-file\") pod \"f818e247-1451-4035-b82d-a56f64d900b2\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " Oct 31 00:49:06.038275 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.037551 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f818e247-1451-4035-b82d-a56f64d900b2-prometheus-data-science-monitoringstack-rulefiles-2" pod "f818e247-1451-4035-b82d-a56f64d900b2" (UID: "f818e247-1451-4035-b82d-a56f64d900b2"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:06.038275 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.037563 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f818e247-1451-4035-b82d-a56f64d900b2-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f818e247-1451-4035-b82d-a56f64d900b2-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"f818e247-1451-4035-b82d-a56f64d900b2\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " Oct 31 00:49:06.038275 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.037620 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f818e247-1451-4035-b82d-a56f64d900b2-config-out\" (UniqueName: \"kubernetes.io/empty-dir/f818e247-1451-4035-b82d-a56f64d900b2-config-out\") pod \"f818e247-1451-4035-b82d-a56f64d900b2\" (UID: \"f818e247-1451-4035-b82d-a56f64d900b2\") " Oct 31 00:49:06.038275 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.037879 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f818e247-1451-4035-b82d-a56f64d900b2-prometheus-data-science-monitoringstack-rulefiles-0" pod "f818e247-1451-4035-b82d-a56f64d900b2" (UID: "f818e247-1451-4035-b82d-a56f64d900b2"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:06.038275 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.037937 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f818e247-1451-4035-b82d-a56f64d900b2-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:06.038275 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.038069 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f818e247-1451-4035-b82d-a56f64d900b2-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:06.039320 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.039043 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f818e247-1451-4035-b82d-a56f64d900b2-prometheus-data-science-monitoringstack-rulefiles-1" pod "f818e247-1451-4035-b82d-a56f64d900b2" (UID: "f818e247-1451-4035-b82d-a56f64d900b2"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:06.039860 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.039828 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85c3b81f-0dad-4b61-ac71-0d1c6737affb-alertmanager-data-science-monitoringstack-db" pod "85c3b81f-0dad-4b61-ac71-0d1c6737affb" (UID: "85c3b81f-0dad-4b61-ac71-0d1c6737affb"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:06.040001 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.039900 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85c3b81f-0dad-4b61-ac71-0d1c6737affb-kube-api-access-xmktw" pod "85c3b81f-0dad-4b61-ac71-0d1c6737affb" (UID: "85c3b81f-0dad-4b61-ac71-0d1c6737affb"). InnerVolumeSpecName "kube-api-access-xmktw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:06.047317 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.047205 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-thanos-prometheus-http-client-file" pod "f818e247-1451-4035-b82d-a56f64d900b2" (UID: "f818e247-1451-4035-b82d-a56f64d900b2"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:06.047317 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.047233 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85c3b81f-0dad-4b61-ac71-0d1c6737affb-config-out" pod "85c3b81f-0dad-4b61-ac71-0d1c6737affb" (UID: "85c3b81f-0dad-4b61-ac71-0d1c6737affb"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:06.047317 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.047268 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f818e247-1451-4035-b82d-a56f64d900b2-kube-api-access-ctfls" pod "f818e247-1451-4035-b82d-a56f64d900b2" (UID: "f818e247-1451-4035-b82d-a56f64d900b2"). InnerVolumeSpecName "kube-api-access-ctfls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:06.047317 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.047280 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-config" pod "f818e247-1451-4035-b82d-a56f64d900b2" (UID: "f818e247-1451-4035-b82d-a56f64d900b2"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:06.047583 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.047316 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "f818e247-1451-4035-b82d-a56f64d900b2" (UID: "f818e247-1451-4035-b82d-a56f64d900b2"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:06.047583 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.047343 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f818e247-1451-4035-b82d-a56f64d900b2-tls-assets" pod "f818e247-1451-4035-b82d-a56f64d900b2" (UID: "f818e247-1451-4035-b82d-a56f64d900b2"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:06.047583 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.047359 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "f818e247-1451-4035-b82d-a56f64d900b2" (UID: "f818e247-1451-4035-b82d-a56f64d900b2"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:06.047583 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.047373 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f818e247-1451-4035-b82d-a56f64d900b2-config-out" pod "f818e247-1451-4035-b82d-a56f64d900b2" (UID: "f818e247-1451-4035-b82d-a56f64d900b2"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:06.047583 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.047413 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85c3b81f-0dad-4b61-ac71-0d1c6737affb-config-volume" pod "85c3b81f-0dad-4b61-ac71-0d1c6737affb" (UID: "85c3b81f-0dad-4b61-ac71-0d1c6737affb"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:06.047583 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.047391 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85c3b81f-0dad-4b61-ac71-0d1c6737affb-tls-assets" pod "85c3b81f-0dad-4b61-ac71-0d1c6737affb" (UID: "85c3b81f-0dad-4b61-ac71-0d1c6737affb"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:06.047583 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.047427 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-secret-prometheus-web-tls-ca" pod "f818e247-1451-4035-b82d-a56f64d900b2" (UID: "f818e247-1451-4035-b82d-a56f64d900b2"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:06.050716 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.050692 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a" pod "f818e247-1451-4035-b82d-a56f64d900b2" (UID: "f818e247-1451-4035-b82d-a56f64d900b2"). InnerVolumeSpecName "pvc-63e34709-8d85-4142-b53a-f655698606ff". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:49:06.052088 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.052063 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85c3b81f-0dad-4b61-ac71-0d1c6737affb-cluster-tls-config" pod "85c3b81f-0dad-4b61-ac71-0d1c6737affb" (UID: "85c3b81f-0dad-4b61-ac71-0d1c6737affb"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:06.061706 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.061687 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85c3b81f-0dad-4b61-ac71-0d1c6737affb-web-config" pod "85c3b81f-0dad-4b61-ac71-0d1c6737affb" (UID: "85c3b81f-0dad-4b61-ac71-0d1c6737affb"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:06.066546 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.066526 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-web-config" pod "f818e247-1451-4035-b82d-a56f64d900b2" (UID: "f818e247-1451-4035-b82d-a56f64d900b2"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:06.123916 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.123895 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:06.139149 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.139129 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5226ae19-6e03-49a3-92f5-8d519cd787ac-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5226ae19-6e03-49a3-92f5-8d519cd787ac-data-science-collector-ta-client-cert\") pod \"5226ae19-6e03-49a3-92f5-8d519cd787ac\" (UID: \"5226ae19-6e03-49a3-92f5-8d519cd787ac\") " Oct 31 00:49:06.139271 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.139173 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5226ae19-6e03-49a3-92f5-8d519cd787ac-tls-certs\" (UniqueName: \"kubernetes.io/secret/5226ae19-6e03-49a3-92f5-8d519cd787ac-tls-certs\") pod \"5226ae19-6e03-49a3-92f5-8d519cd787ac\" (UID: \"5226ae19-6e03-49a3-92f5-8d519cd787ac\") " Oct 31 00:49:06.139271 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.139190 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5226ae19-6e03-49a3-92f5-8d519cd787ac-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5226ae19-6e03-49a3-92f5-8d519cd787ac-otlp-tls-certs\") pod \"5226ae19-6e03-49a3-92f5-8d519cd787ac\" (UID: \"5226ae19-6e03-49a3-92f5-8d519cd787ac\") " Oct 31 00:49:06.139271 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.139207 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5226ae19-6e03-49a3-92f5-8d519cd787ac-otc-internal\" (UniqueName: \"kubernetes.io/configmap/5226ae19-6e03-49a3-92f5-8d519cd787ac-otc-internal\") pod \"5226ae19-6e03-49a3-92f5-8d519cd787ac\" (UID: \"5226ae19-6e03-49a3-92f5-8d519cd787ac\") " Oct 31 00:49:06.139271 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.139259 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5226ae19-6e03-49a3-92f5-8d519cd787ac-kube-api-access-vxnpt\" (UniqueName: \"kubernetes.io/projected/5226ae19-6e03-49a3-92f5-8d519cd787ac-kube-api-access-vxnpt\") pod \"5226ae19-6e03-49a3-92f5-8d519cd787ac\" (UID: \"5226ae19-6e03-49a3-92f5-8d519cd787ac\") " Oct 31 00:49:06.139489 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.139319 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5226ae19-6e03-49a3-92f5-8d519cd787ac-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5226ae19-6e03-49a3-92f5-8d519cd787ac-monitoring-tls-certs\") pod \"5226ae19-6e03-49a3-92f5-8d519cd787ac\" (UID: \"5226ae19-6e03-49a3-92f5-8d519cd787ac\") " Oct 31 00:49:06.139607 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.139558 5274 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:06.139607 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.139578 5274 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-web-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:06.139607 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.139592 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:06.139607 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.139605 5274 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/85c3b81f-0dad-4b61-ac71-0d1c6737affb-tls-assets\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:06.139811 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.139620 5274 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:06.139811 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.139635 5274 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f818e247-1451-4035-b82d-a56f64d900b2-config-out\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:06.139811 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.139644 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5226ae19-6e03-49a3-92f5-8d519cd787ac-otc-internal" pod "5226ae19-6e03-49a3-92f5-8d519cd787ac" (UID: "5226ae19-6e03-49a3-92f5-8d519cd787ac"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:06.139811 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.139671 5274 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" " Oct 31 00:49:06.139811 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.139686 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmktw\" (UniqueName: \"kubernetes.io/projected/85c3b81f-0dad-4b61-ac71-0d1c6737affb-kube-api-access-xmktw\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:06.139811 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.139699 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f818e247-1451-4035-b82d-a56f64d900b2-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:06.139811 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.139714 5274 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/85c3b81f-0dad-4b61-ac71-0d1c6737affb-web-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:06.139811 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.139727 5274 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/85c3b81f-0dad-4b61-ac71-0d1c6737affb-cluster-tls-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:06.139811 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.139742 5274 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/85c3b81f-0dad-4b61-ac71-0d1c6737affb-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:06.139811 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.139757 5274 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-secret-prometheus-web-tls-ca\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:06.139811 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.139769 5274 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/85c3b81f-0dad-4b61-ac71-0d1c6737affb-config-volume\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:06.139811 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.139778 5274 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f818e247-1451-4035-b82d-a56f64d900b2-tls-assets\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:06.139811 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.139786 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctfls\" (UniqueName: \"kubernetes.io/projected/f818e247-1451-4035-b82d-a56f64d900b2-kube-api-access-ctfls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:06.139811 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.139794 5274 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/85c3b81f-0dad-4b61-ac71-0d1c6737affb-config-out\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:06.139811 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.139803 5274 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f818e247-1451-4035-b82d-a56f64d900b2-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:06.141408 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.141385 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5226ae19-6e03-49a3-92f5-8d519cd787ac-otlp-tls-certs" pod "5226ae19-6e03-49a3-92f5-8d519cd787ac" (UID: "5226ae19-6e03-49a3-92f5-8d519cd787ac"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:06.141798 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.141774 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5226ae19-6e03-49a3-92f5-8d519cd787ac-kube-api-access-vxnpt" pod "5226ae19-6e03-49a3-92f5-8d519cd787ac" (UID: "5226ae19-6e03-49a3-92f5-8d519cd787ac"). InnerVolumeSpecName "kube-api-access-vxnpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:06.141874 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.141779 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5226ae19-6e03-49a3-92f5-8d519cd787ac-data-science-collector-ta-client-cert" pod "5226ae19-6e03-49a3-92f5-8d519cd787ac" (UID: "5226ae19-6e03-49a3-92f5-8d519cd787ac"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:06.141874 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.141829 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5226ae19-6e03-49a3-92f5-8d519cd787ac-monitoring-tls-certs" pod "5226ae19-6e03-49a3-92f5-8d519cd787ac" (UID: "5226ae19-6e03-49a3-92f5-8d519cd787ac"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:06.141874 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.141850 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5226ae19-6e03-49a3-92f5-8d519cd787ac-tls-certs" pod "5226ae19-6e03-49a3-92f5-8d519cd787ac" (UID: "5226ae19-6e03-49a3-92f5-8d519cd787ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:06.152246 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.152231 5274 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-63e34709-8d85-4142-b53a-f655698606ff" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a") on node "ip-10-0-129-70.us-east-2.compute.internal" Oct 31 00:49:06.240949 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.240889 5274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5226ae19-6e03-49a3-92f5-8d519cd787ac-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:06.240949 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.240914 5274 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5226ae19-6e03-49a3-92f5-8d519cd787ac-otlp-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:06.240949 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.240928 5274 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5226ae19-6e03-49a3-92f5-8d519cd787ac-otc-internal\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:06.240949 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.240941 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxnpt\" (UniqueName: \"kubernetes.io/projected/5226ae19-6e03-49a3-92f5-8d519cd787ac-kube-api-access-vxnpt\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:06.241163 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.240977 5274 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5226ae19-6e03-49a3-92f5-8d519cd787ac-monitoring-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:06.241163 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.240993 5274 reconciler_common.go:299] "Volume detached for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"csi-3beb91b3ebf177571ca474589da3560cd18787cfa50b2a0b9bfef85661e6cf21\"" Oct 31 00:49:06.241163 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.241006 5274 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5226ae19-6e03-49a3-92f5-8d519cd787ac-data-science-collector-ta-client-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:06.882313 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.882281 5274 generic.go:356] "Generic (PLEG): container finished" podID="85c3b81f-0dad-4b61-ac71-0d1c6737affb" containerID="17cd760deb2cd9d0468e9513e819a9503e407cdfc2093370f22cb5283541fe4d" exitCode=0 Oct 31 00:49:06.882445 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.882339 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:49:06.882445 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.882363 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"85c3b81f-0dad-4b61-ac71-0d1c6737affb","Type":"ContainerDied","Data":"17cd760deb2cd9d0468e9513e819a9503e407cdfc2093370f22cb5283541fe4d"} Oct 31 00:49:06.882445 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.882433 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"85c3b81f-0dad-4b61-ac71-0d1c6737affb","Type":"ContainerDied","Data":"242ef08e5c564b92297e9247718454c9f2b30bc056ed4fc44d24de9daa25fe85"} Oct 31 00:49:06.882580 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.882472 5274 scope.go:122] "RemoveContainer" containerID="fd069024b262e52e595628051b259cd3d6ea2529b88b18fd5f597f1adc43b72a" Oct 31 00:49:06.885818 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.885793 5274 generic.go:356] "Generic (PLEG): container finished" podID="5226ae19-6e03-49a3-92f5-8d519cd787ac" containerID="769f4aed06d094eac3ded0c8199231dea1dd73ab5137e35c1503817cc8d35e98" exitCode=0 Oct 31 00:49:06.885931 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.885851 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:06.885931 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.885867 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5226ae19-6e03-49a3-92f5-8d519cd787ac","Type":"ContainerDied","Data":"769f4aed06d094eac3ded0c8199231dea1dd73ab5137e35c1503817cc8d35e98"} Oct 31 00:49:06.885931 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.885909 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5226ae19-6e03-49a3-92f5-8d519cd787ac","Type":"ContainerDied","Data":"dc94f80334b1e37708686cd0be51a03574cc7ae50766d002af86055df2c51966"} Oct 31 00:49:06.886149 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.886136 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:06.892087 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.892067 5274 scope.go:122] "RemoveContainer" containerID="17cd760deb2cd9d0468e9513e819a9503e407cdfc2093370f22cb5283541fe4d" Oct 31 00:49:06.901412 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.901396 5274 scope.go:122] "RemoveContainer" containerID="ec50354b3c9281e211357ff125cdb4e7e5be4b33453ff15e97a3cd9fe29a665b" Oct 31 00:49:06.908949 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.908933 5274 scope.go:122] "RemoveContainer" containerID="fd069024b262e52e595628051b259cd3d6ea2529b88b18fd5f597f1adc43b72a" Oct 31 00:49:06.909251 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:49:06.909233 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd069024b262e52e595628051b259cd3d6ea2529b88b18fd5f597f1adc43b72a\": container with ID starting with fd069024b262e52e595628051b259cd3d6ea2529b88b18fd5f597f1adc43b72a not found: ID does not exist" containerID="fd069024b262e52e595628051b259cd3d6ea2529b88b18fd5f597f1adc43b72a" Oct 31 00:49:06.909300 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.909258 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd069024b262e52e595628051b259cd3d6ea2529b88b18fd5f597f1adc43b72a"} err="failed to get container status \"fd069024b262e52e595628051b259cd3d6ea2529b88b18fd5f597f1adc43b72a\": rpc error: code = NotFound desc = could not find container \"fd069024b262e52e595628051b259cd3d6ea2529b88b18fd5f597f1adc43b72a\": container with ID starting with fd069024b262e52e595628051b259cd3d6ea2529b88b18fd5f597f1adc43b72a not found: ID does not exist" Oct 31 00:49:06.909300 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.909274 5274 scope.go:122] "RemoveContainer" containerID="17cd760deb2cd9d0468e9513e819a9503e407cdfc2093370f22cb5283541fe4d" Oct 31 00:49:06.909485 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:49:06.909471 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17cd760deb2cd9d0468e9513e819a9503e407cdfc2093370f22cb5283541fe4d\": container with ID starting with 17cd760deb2cd9d0468e9513e819a9503e407cdfc2093370f22cb5283541fe4d not found: ID does not exist" containerID="17cd760deb2cd9d0468e9513e819a9503e407cdfc2093370f22cb5283541fe4d" Oct 31 00:49:06.909521 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.909489 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17cd760deb2cd9d0468e9513e819a9503e407cdfc2093370f22cb5283541fe4d"} err="failed to get container status \"17cd760deb2cd9d0468e9513e819a9503e407cdfc2093370f22cb5283541fe4d\": rpc error: code = NotFound desc = could not find container \"17cd760deb2cd9d0468e9513e819a9503e407cdfc2093370f22cb5283541fe4d\": container with ID starting with 17cd760deb2cd9d0468e9513e819a9503e407cdfc2093370f22cb5283541fe4d not found: ID does not exist" Oct 31 00:49:06.909521 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.909501 5274 scope.go:122] "RemoveContainer" containerID="ec50354b3c9281e211357ff125cdb4e7e5be4b33453ff15e97a3cd9fe29a665b" Oct 31 00:49:06.909710 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:49:06.909695 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec50354b3c9281e211357ff125cdb4e7e5be4b33453ff15e97a3cd9fe29a665b\": container with ID starting with ec50354b3c9281e211357ff125cdb4e7e5be4b33453ff15e97a3cd9fe29a665b not found: ID does not exist" containerID="ec50354b3c9281e211357ff125cdb4e7e5be4b33453ff15e97a3cd9fe29a665b" Oct 31 00:49:06.909748 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.909713 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec50354b3c9281e211357ff125cdb4e7e5be4b33453ff15e97a3cd9fe29a665b"} err="failed to get container status \"ec50354b3c9281e211357ff125cdb4e7e5be4b33453ff15e97a3cd9fe29a665b\": rpc error: code = NotFound desc = could not find container \"ec50354b3c9281e211357ff125cdb4e7e5be4b33453ff15e97a3cd9fe29a665b\": container with ID starting with ec50354b3c9281e211357ff125cdb4e7e5be4b33453ff15e97a3cd9fe29a665b not found: ID does not exist" Oct 31 00:49:06.909748 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.909726 5274 scope.go:122] "RemoveContainer" containerID="769f4aed06d094eac3ded0c8199231dea1dd73ab5137e35c1503817cc8d35e98" Oct 31 00:49:06.918119 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.918099 5274 scope.go:122] "RemoveContainer" containerID="769f4aed06d094eac3ded0c8199231dea1dd73ab5137e35c1503817cc8d35e98" Oct 31 00:49:06.918342 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.918324 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:49:06.918399 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:49:06.918356 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"769f4aed06d094eac3ded0c8199231dea1dd73ab5137e35c1503817cc8d35e98\": container with ID starting with 769f4aed06d094eac3ded0c8199231dea1dd73ab5137e35c1503817cc8d35e98 not found: ID does not exist" containerID="769f4aed06d094eac3ded0c8199231dea1dd73ab5137e35c1503817cc8d35e98" Oct 31 00:49:06.918399 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.918380 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"769f4aed06d094eac3ded0c8199231dea1dd73ab5137e35c1503817cc8d35e98"} err="failed to get container status \"769f4aed06d094eac3ded0c8199231dea1dd73ab5137e35c1503817cc8d35e98\": rpc error: code = NotFound desc = could not find container \"769f4aed06d094eac3ded0c8199231dea1dd73ab5137e35c1503817cc8d35e98\": container with ID starting with 769f4aed06d094eac3ded0c8199231dea1dd73ab5137e35c1503817cc8d35e98 not found: ID does not exist" Oct 31 00:49:06.922846 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.922827 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:49:06.934533 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.934513 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:49:06.939378 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.939360 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:49:06.952996 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.952978 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:49:06.959796 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:06.959779 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:49:07.481808 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:07.481775 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45d15e31-0f2f-4c64-b35b-c133895ccea5" path="/var/lib/kubelet/pods/45d15e31-0f2f-4c64-b35b-c133895ccea5/volumes" Oct 31 00:49:07.482240 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:07.482185 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5226ae19-6e03-49a3-92f5-8d519cd787ac" path="/var/lib/kubelet/pods/5226ae19-6e03-49a3-92f5-8d519cd787ac/volumes" Oct 31 00:49:07.482526 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:07.482512 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85c3b81f-0dad-4b61-ac71-0d1c6737affb" path="/var/lib/kubelet/pods/85c3b81f-0dad-4b61-ac71-0d1c6737affb/volumes" Oct 31 00:49:07.483340 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:07.483325 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e662ea68-3516-4efd-8e66-f86532b11225" path="/var/lib/kubelet/pods/e662ea68-3516-4efd-8e66-f86532b11225/volumes" Oct 31 00:49:07.483709 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:07.483698 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f818e247-1451-4035-b82d-a56f64d900b2" path="/var/lib/kubelet/pods/f818e247-1451-4035-b82d-a56f64d900b2/volumes" Oct 31 00:49:17.149567 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.149537 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-p2hvj"] Oct 31 00:49:17.150062 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.149834 5274 cpu_manager.go:414] "containerMap: removing container" podUID="45d15e31-0f2f-4c64-b35b-c133895ccea5" containerName="kube-rbac-proxy" Oct 31 00:49:17.150062 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.149844 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45d15e31-0f2f-4c64-b35b-c133895ccea5" containerName="kube-rbac-proxy" Oct 31 00:49:17.150062 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.149855 5274 cpu_manager.go:414] "containerMap: removing container" podUID="85c3b81f-0dad-4b61-ac71-0d1c6737affb" containerName="config-reloader" Oct 31 00:49:17.150062 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.149860 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85c3b81f-0dad-4b61-ac71-0d1c6737affb" containerName="config-reloader" Oct 31 00:49:17.150062 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.149870 5274 cpu_manager.go:414] "containerMap: removing container" podUID="f818e247-1451-4035-b82d-a56f64d900b2" containerName="prometheus" Oct 31 00:49:17.150062 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.149874 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f818e247-1451-4035-b82d-a56f64d900b2" containerName="prometheus" Oct 31 00:49:17.150062 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.149886 5274 cpu_manager.go:414] "containerMap: removing container" podUID="85c3b81f-0dad-4b61-ac71-0d1c6737affb" containerName="init-config-reloader" Oct 31 00:49:17.150062 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.149890 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85c3b81f-0dad-4b61-ac71-0d1c6737affb" containerName="init-config-reloader" Oct 31 00:49:17.150062 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.149896 5274 cpu_manager.go:414] "containerMap: removing container" podUID="85c3b81f-0dad-4b61-ac71-0d1c6737affb" containerName="alertmanager" Oct 31 00:49:17.150062 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.149900 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85c3b81f-0dad-4b61-ac71-0d1c6737affb" containerName="alertmanager" Oct 31 00:49:17.150062 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.149906 5274 cpu_manager.go:414] "containerMap: removing container" podUID="5226ae19-6e03-49a3-92f5-8d519cd787ac" containerName="otc-container" Oct 31 00:49:17.150062 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.149910 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5226ae19-6e03-49a3-92f5-8d519cd787ac" containerName="otc-container" Oct 31 00:49:17.150062 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.149921 5274 cpu_manager.go:414] "containerMap: removing container" podUID="e662ea68-3516-4efd-8e66-f86532b11225" containerName="korrel8r" Oct 31 00:49:17.150062 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.149925 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e662ea68-3516-4efd-8e66-f86532b11225" containerName="korrel8r" Oct 31 00:49:17.150062 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.149930 5274 cpu_manager.go:414] "containerMap: removing container" podUID="f818e247-1451-4035-b82d-a56f64d900b2" containerName="thanos-sidecar" Oct 31 00:49:17.150062 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.149935 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f818e247-1451-4035-b82d-a56f64d900b2" containerName="thanos-sidecar" Oct 31 00:49:17.150062 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.149947 5274 cpu_manager.go:414] "containerMap: removing container" podUID="f818e247-1451-4035-b82d-a56f64d900b2" containerName="init-config-reloader" Oct 31 00:49:17.150062 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.149972 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f818e247-1451-4035-b82d-a56f64d900b2" containerName="init-config-reloader" Oct 31 00:49:17.150062 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.149989 5274 cpu_manager.go:414] "containerMap: removing container" podUID="f818e247-1451-4035-b82d-a56f64d900b2" containerName="config-reloader" Oct 31 00:49:17.150062 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.149993 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f818e247-1451-4035-b82d-a56f64d900b2" containerName="config-reloader" Oct 31 00:49:17.150062 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.150066 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="f818e247-1451-4035-b82d-a56f64d900b2" containerName="prometheus" Oct 31 00:49:17.150840 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.150079 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="5226ae19-6e03-49a3-92f5-8d519cd787ac" containerName="otc-container" Oct 31 00:49:17.150840 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.150086 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="85c3b81f-0dad-4b61-ac71-0d1c6737affb" containerName="alertmanager" Oct 31 00:49:17.150840 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.150092 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="e662ea68-3516-4efd-8e66-f86532b11225" containerName="korrel8r" Oct 31 00:49:17.150840 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.150097 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="f818e247-1451-4035-b82d-a56f64d900b2" containerName="thanos-sidecar" Oct 31 00:49:17.150840 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.150105 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="f818e247-1451-4035-b82d-a56f64d900b2" containerName="config-reloader" Oct 31 00:49:17.150840 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.150111 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="45d15e31-0f2f-4c64-b35b-c133895ccea5" containerName="kube-rbac-proxy" Oct 31 00:49:17.150840 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.150116 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="85c3b81f-0dad-4b61-ac71-0d1c6737affb" containerName="config-reloader" Oct 31 00:49:17.154414 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.154391 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-p2hvj" Oct 31 00:49:17.160915 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.160892 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\"" Oct 31 00:49:17.164963 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.164915 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:49:17.165103 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.164997 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:49:17.165103 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.164997 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:49:17.166503 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.166484 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-p2hvj"] Oct 31 00:49:17.213105 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.213079 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-499qg"] Oct 31 00:49:17.216570 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.216553 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-499qg" Oct 31 00:49:17.222610 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.222583 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cct76\" (UniqueName: \"kubernetes.io/projected/149823e9-b0e4-476c-9d37-d4581e7ea7d1-kube-api-access-cct76\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-p2hvj\" (UID: \"149823e9-b0e4-476c-9d37-d4581e7ea7d1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-p2hvj" Oct 31 00:49:17.222731 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.222646 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/149823e9-b0e4-476c-9d37-d4581e7ea7d1-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-p2hvj\" (UID: \"149823e9-b0e4-476c-9d37-d4581e7ea7d1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-p2hvj" Oct 31 00:49:17.222912 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.222752 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/149823e9-b0e4-476c-9d37-d4581e7ea7d1-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-p2hvj\" (UID: \"149823e9-b0e4-476c-9d37-d4581e7ea7d1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-p2hvj" Oct 31 00:49:17.222912 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.222782 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/149823e9-b0e4-476c-9d37-d4581e7ea7d1-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-p2hvj\" (UID: \"149823e9-b0e4-476c-9d37-d4581e7ea7d1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-p2hvj" Oct 31 00:49:17.222912 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.222873 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/149823e9-b0e4-476c-9d37-d4581e7ea7d1-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-p2hvj\" (UID: \"149823e9-b0e4-476c-9d37-d4581e7ea7d1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-p2hvj" Oct 31 00:49:17.226829 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.226808 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-499qg"] Oct 31 00:49:17.233580 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.233537 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\"" Oct 31 00:49:17.233580 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.233556 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 31 00:49:17.233742 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.233625 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-5vd4s\"" Oct 31 00:49:17.233926 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.233906 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 31 00:49:17.234062 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.234007 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-8lvmj\"" Oct 31 00:49:17.264168 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.264152 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 31 00:49:17.323521 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.323500 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/149823e9-b0e4-476c-9d37-d4581e7ea7d1-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-p2hvj\" (UID: \"149823e9-b0e4-476c-9d37-d4581e7ea7d1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-p2hvj" Oct 31 00:49:17.323624 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.323526 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/149823e9-b0e4-476c-9d37-d4581e7ea7d1-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-p2hvj\" (UID: \"149823e9-b0e4-476c-9d37-d4581e7ea7d1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-p2hvj" Oct 31 00:49:17.323624 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.323562 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/149823e9-b0e4-476c-9d37-d4581e7ea7d1-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-p2hvj\" (UID: \"149823e9-b0e4-476c-9d37-d4581e7ea7d1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-p2hvj" Oct 31 00:49:17.323624 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.323601 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cct76\" (UniqueName: \"kubernetes.io/projected/149823e9-b0e4-476c-9d37-d4581e7ea7d1-kube-api-access-cct76\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-p2hvj\" (UID: \"149823e9-b0e4-476c-9d37-d4581e7ea7d1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-p2hvj" Oct 31 00:49:17.323792 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.323625 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67m5x\" (UniqueName: \"kubernetes.io/projected/58d39fb2-6f91-4f59-aa7c-45e69e43cb12-kube-api-access-67m5x\") pod \"korrel8r-5cbcc89b8f-499qg\" (UID: \"58d39fb2-6f91-4f59-aa7c-45e69e43cb12\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-499qg" Oct 31 00:49:17.323792 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.323650 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/149823e9-b0e4-476c-9d37-d4581e7ea7d1-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-p2hvj\" (UID: \"149823e9-b0e4-476c-9d37-d4581e7ea7d1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-p2hvj" Oct 31 00:49:17.323792 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.323667 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/58d39fb2-6f91-4f59-aa7c-45e69e43cb12-config\") pod \"korrel8r-5cbcc89b8f-499qg\" (UID: \"58d39fb2-6f91-4f59-aa7c-45e69e43cb12\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-499qg" Oct 31 00:49:17.323792 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.323688 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/58d39fb2-6f91-4f59-aa7c-45e69e43cb12-tls\") pod \"korrel8r-5cbcc89b8f-499qg\" (UID: \"58d39fb2-6f91-4f59-aa7c-45e69e43cb12\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-499qg" Oct 31 00:49:17.324306 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.324288 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/149823e9-b0e4-476c-9d37-d4581e7ea7d1-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-p2hvj\" (UID: \"149823e9-b0e4-476c-9d37-d4581e7ea7d1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-p2hvj" Oct 31 00:49:17.326027 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.325984 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/149823e9-b0e4-476c-9d37-d4581e7ea7d1-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-p2hvj\" (UID: \"149823e9-b0e4-476c-9d37-d4581e7ea7d1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-p2hvj" Oct 31 00:49:17.326027 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.325995 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/149823e9-b0e4-476c-9d37-d4581e7ea7d1-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-p2hvj\" (UID: \"149823e9-b0e4-476c-9d37-d4581e7ea7d1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-p2hvj" Oct 31 00:49:17.326188 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.326132 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/149823e9-b0e4-476c-9d37-d4581e7ea7d1-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-p2hvj\" (UID: \"149823e9-b0e4-476c-9d37-d4581e7ea7d1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-p2hvj" Oct 31 00:49:17.335966 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.335939 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cct76\" (UniqueName: \"kubernetes.io/projected/149823e9-b0e4-476c-9d37-d4581e7ea7d1-kube-api-access-cct76\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-p2hvj\" (UID: \"149823e9-b0e4-476c-9d37-d4581e7ea7d1\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-p2hvj" Oct 31 00:49:17.424895 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.424828 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67m5x\" (UniqueName: \"kubernetes.io/projected/58d39fb2-6f91-4f59-aa7c-45e69e43cb12-kube-api-access-67m5x\") pod \"korrel8r-5cbcc89b8f-499qg\" (UID: \"58d39fb2-6f91-4f59-aa7c-45e69e43cb12\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-499qg" Oct 31 00:49:17.424895 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.424870 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/58d39fb2-6f91-4f59-aa7c-45e69e43cb12-config\") pod \"korrel8r-5cbcc89b8f-499qg\" (UID: \"58d39fb2-6f91-4f59-aa7c-45e69e43cb12\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-499qg" Oct 31 00:49:17.424895 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.424885 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/58d39fb2-6f91-4f59-aa7c-45e69e43cb12-tls\") pod \"korrel8r-5cbcc89b8f-499qg\" (UID: \"58d39fb2-6f91-4f59-aa7c-45e69e43cb12\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-499qg" Oct 31 00:49:17.425633 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.425611 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/58d39fb2-6f91-4f59-aa7c-45e69e43cb12-config\") pod \"korrel8r-5cbcc89b8f-499qg\" (UID: \"58d39fb2-6f91-4f59-aa7c-45e69e43cb12\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-499qg" Oct 31 00:49:17.427135 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.427119 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/58d39fb2-6f91-4f59-aa7c-45e69e43cb12-tls\") pod \"korrel8r-5cbcc89b8f-499qg\" (UID: \"58d39fb2-6f91-4f59-aa7c-45e69e43cb12\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-499qg" Oct 31 00:49:17.437011 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.436988 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67m5x\" (UniqueName: \"kubernetes.io/projected/58d39fb2-6f91-4f59-aa7c-45e69e43cb12-kube-api-access-67m5x\") pod \"korrel8r-5cbcc89b8f-499qg\" (UID: \"58d39fb2-6f91-4f59-aa7c-45e69e43cb12\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-499qg" Oct 31 00:49:17.467973 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.467937 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-p2hvj" Oct 31 00:49:17.530193 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.529875 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-499qg" Oct 31 00:49:17.684306 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.684238 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:49:17.688881 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.688857 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:17.698739 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.698161 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:49:17.698739 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.698253 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:49:17.698739 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.698366 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:49:17.699124 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.699101 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:49:17.699254 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.699188 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:49:17.700434 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.700415 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:49:17.728189 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.728157 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6b16f028-c18a-4c4d-9a88-d6c961bfa589-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"6b16f028-c18a-4c4d-9a88-d6c961bfa589\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:17.728461 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.728437 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6b16f028-c18a-4c4d-9a88-d6c961bfa589-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"6b16f028-c18a-4c4d-9a88-d6c961bfa589\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:17.728593 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.728575 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b16f028-c18a-4c4d-9a88-d6c961bfa589-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6b16f028-c18a-4c4d-9a88-d6c961bfa589\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:17.728774 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.728756 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9pz6\" (UniqueName: \"kubernetes.io/projected/6b16f028-c18a-4c4d-9a88-d6c961bfa589-kube-api-access-b9pz6\") pod \"data-science-collector-collector-0\" (UID: \"6b16f028-c18a-4c4d-9a88-d6c961bfa589\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:17.729223 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.729202 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6b16f028-c18a-4c4d-9a88-d6c961bfa589-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6b16f028-c18a-4c4d-9a88-d6c961bfa589\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:17.729496 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.729478 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6b16f028-c18a-4c4d-9a88-d6c961bfa589-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6b16f028-c18a-4c4d-9a88-d6c961bfa589\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:17.760196 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.760024 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-p2hvj"] Oct 31 00:49:17.761713 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:49:17.761684 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod149823e9_b0e4_476c_9d37_d4581e7ea7d1.slice/crio-ea358c824d5748fcb9cb2cc4c3c1f2d6daf1dadcc49a634693b22a6bb3366ebf WatchSource:0}: Error finding container ea358c824d5748fcb9cb2cc4c3c1f2d6daf1dadcc49a634693b22a6bb3366ebf: Status 404 returned error can't find the container with id ea358c824d5748fcb9cb2cc4c3c1f2d6daf1dadcc49a634693b22a6bb3366ebf Oct 31 00:49:17.813266 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.812817 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-499qg"] Oct 31 00:49:17.815620 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:49:17.815594 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58d39fb2_6f91_4f59_aa7c_45e69e43cb12.slice/crio-6681a5641fc7b1d4f8726a539850356defe88b71a608522ae2841273e030e485 WatchSource:0}: Error finding container 6681a5641fc7b1d4f8726a539850356defe88b71a608522ae2841273e030e485: Status 404 returned error can't find the container with id 6681a5641fc7b1d4f8726a539850356defe88b71a608522ae2841273e030e485 Oct 31 00:49:17.831109 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.831085 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6b16f028-c18a-4c4d-9a88-d6c961bfa589-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6b16f028-c18a-4c4d-9a88-d6c961bfa589\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:17.831288 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.831270 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6b16f028-c18a-4c4d-9a88-d6c961bfa589-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6b16f028-c18a-4c4d-9a88-d6c961bfa589\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:17.831501 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.831457 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6b16f028-c18a-4c4d-9a88-d6c961bfa589-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"6b16f028-c18a-4c4d-9a88-d6c961bfa589\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:17.831652 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:49:17.831580 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found Oct 31 00:49:17.831759 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:49:17.831671 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b16f028-c18a-4c4d-9a88-d6c961bfa589-otlp-tls-certs podName:6b16f028-c18a-4c4d-9a88-d6c961bfa589 nodeName:}" failed. No retries permitted until 2026-10-31 00:49:18.331651842 +0000 UTC m=+2994.968470550 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/6b16f028-c18a-4c4d-9a88-d6c961bfa589-otlp-tls-certs") pod "data-science-collector-collector-0" (UID: "6b16f028-c18a-4c4d-9a88-d6c961bfa589") : secret "data-science-collector-collector-headless-tls" not found Oct 31 00:49:17.832840 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.832095 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6b16f028-c18a-4c4d-9a88-d6c961bfa589-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"6b16f028-c18a-4c4d-9a88-d6c961bfa589\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:17.832840 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.832143 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b16f028-c18a-4c4d-9a88-d6c961bfa589-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6b16f028-c18a-4c4d-9a88-d6c961bfa589\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:17.832840 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.832174 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9pz6\" (UniqueName: \"kubernetes.io/projected/6b16f028-c18a-4c4d-9a88-d6c961bfa589-kube-api-access-b9pz6\") pod \"data-science-collector-collector-0\" (UID: \"6b16f028-c18a-4c4d-9a88-d6c961bfa589\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:17.832840 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.832797 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6b16f028-c18a-4c4d-9a88-d6c961bfa589-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"6b16f028-c18a-4c4d-9a88-d6c961bfa589\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:17.834363 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.834341 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6b16f028-c18a-4c4d-9a88-d6c961bfa589-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6b16f028-c18a-4c4d-9a88-d6c961bfa589\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:17.834687 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.834665 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6b16f028-c18a-4c4d-9a88-d6c961bfa589-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"6b16f028-c18a-4c4d-9a88-d6c961bfa589\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:17.835290 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.835271 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b16f028-c18a-4c4d-9a88-d6c961bfa589-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6b16f028-c18a-4c4d-9a88-d6c961bfa589\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:17.845062 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.845041 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9pz6\" (UniqueName: \"kubernetes.io/projected/6b16f028-c18a-4c4d-9a88-d6c961bfa589-kube-api-access-b9pz6\") pod \"data-science-collector-collector-0\" (UID: \"6b16f028-c18a-4c4d-9a88-d6c961bfa589\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:17.949867 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.949793 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-499qg" event={"ID":"58d39fb2-6f91-4f59-aa7c-45e69e43cb12","Type":"ContainerStarted","Data":"fb76b61862f6ca9f45a389d10cfe5da76ea8b6356f3c4e58b5c93118b7e1298d"} Oct 31 00:49:17.949867 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.949845 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-499qg" event={"ID":"58d39fb2-6f91-4f59-aa7c-45e69e43cb12","Type":"ContainerStarted","Data":"6681a5641fc7b1d4f8726a539850356defe88b71a608522ae2841273e030e485"} Oct 31 00:49:17.950131 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.949935 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-499qg" Oct 31 00:49:17.951230 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.951193 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-p2hvj" event={"ID":"149823e9-b0e4-476c-9d37-d4581e7ea7d1","Type":"ContainerStarted","Data":"ea358c824d5748fcb9cb2cc4c3c1f2d6daf1dadcc49a634693b22a6bb3366ebf"} Oct 31 00:49:17.968823 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:17.968780 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-499qg" podStartSLOduration=0.968767826 podStartE2EDuration="968.767826ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:17.967778925 +0000 UTC m=+2994.604597642" watchObservedRunningTime="2026-10-31 00:49:17.968767826 +0000 UTC m=+2994.605586541" Oct 31 00:49:18.335996 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:18.335933 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6b16f028-c18a-4c4d-9a88-d6c961bfa589-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6b16f028-c18a-4c4d-9a88-d6c961bfa589\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:18.336350 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:49:18.336069 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found Oct 31 00:49:18.336350 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:49:18.336127 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b16f028-c18a-4c4d-9a88-d6c961bfa589-otlp-tls-certs podName:6b16f028-c18a-4c4d-9a88-d6c961bfa589 nodeName:}" failed. No retries permitted until 2026-10-31 00:49:19.336113813 +0000 UTC m=+2995.972932529 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/6b16f028-c18a-4c4d-9a88-d6c961bfa589-otlp-tls-certs") pod "data-science-collector-collector-0" (UID: "6b16f028-c18a-4c4d-9a88-d6c961bfa589") : secret "data-science-collector-collector-headless-tls" not found Oct 31 00:49:18.957901 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:18.957865 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-p2hvj" event={"ID":"149823e9-b0e4-476c-9d37-d4581e7ea7d1","Type":"ContainerStarted","Data":"edb61927ba6edbbfebed49df327d5d7d7aaa15c5dc32ffedb9258645f93e5db8"} Oct 31 00:49:18.958151 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:18.957945 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-p2hvj" Oct 31 00:49:18.978788 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:18.978749 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-p2hvj" podStartSLOduration=1.978737964 podStartE2EDuration="1.978737964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:18.977148365 +0000 UTC m=+2995.613967085" watchObservedRunningTime="2026-10-31 00:49:18.978737964 +0000 UTC m=+2995.615556680" Oct 31 00:49:19.346309 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:19.346232 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6b16f028-c18a-4c4d-9a88-d6c961bfa589-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6b16f028-c18a-4c4d-9a88-d6c961bfa589\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:19.348575 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:19.348556 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6b16f028-c18a-4c4d-9a88-d6c961bfa589-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"6b16f028-c18a-4c4d-9a88-d6c961bfa589\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:19.506280 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:19.506249 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:19.784099 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:19.784075 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:49:19.786077 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:49:19.786054 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b16f028_c18a_4c4d_9a88_d6c961bfa589.slice/crio-29f5553c23725386d94aaa8a8ba09c29f1b1634edd02ce4ba058456e0abb83e5 WatchSource:0}: Error finding container 29f5553c23725386d94aaa8a8ba09c29f1b1634edd02ce4ba058456e0abb83e5: Status 404 returned error can't find the container with id 29f5553c23725386d94aaa8a8ba09c29f1b1634edd02ce4ba058456e0abb83e5 Oct 31 00:49:19.788685 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:19.788653 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:49:19.788763 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:19.788708 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:49:19.964750 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:19.964717 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"6b16f028-c18a-4c4d-9a88-d6c961bfa589","Type":"ContainerStarted","Data":"001b2d9b6da75c3217fca9993f39291f45c0bdcdddc7fb091bc7beefaa92ae7c"} Oct 31 00:49:19.964914 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:19.964768 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"6b16f028-c18a-4c4d-9a88-d6c961bfa589","Type":"ContainerStarted","Data":"29f5553c23725386d94aaa8a8ba09c29f1b1634edd02ce4ba058456e0abb83e5"} Oct 31 00:49:19.995315 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:19.995262 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.9952490210000002 podStartE2EDuration="2.995249021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:19.992775082 +0000 UTC m=+2996.629593797" watchObservedRunningTime="2026-10-31 00:49:19.995249021 +0000 UTC m=+2996.632067735" Oct 31 00:49:20.898570 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:49:20.898522 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:49:23.842639 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:23.842608 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:49:23.846567 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:23.846538 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:49:23.862129 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:23.862109 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:49:23.876313 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:23.876297 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 31 00:49:23.876397 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:23.876329 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 31 00:49:23.876397 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:23.876363 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 31 00:49:23.876397 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:23.876304 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-wzs48\"" Oct 31 00:49:23.876599 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:23.876583 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:49:23.880776 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:23.880758 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9777d542-77d4-4ad3-b2c5-a60b8ec24d68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:49:23.880875 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:23.880802 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kgj5\" (UniqueName: \"kubernetes.io/projected/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-kube-api-access-4kgj5\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9777d542-77d4-4ad3-b2c5-a60b8ec24d68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:49:23.880929 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:23.880898 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9777d542-77d4-4ad3-b2c5-a60b8ec24d68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:49:23.881006 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:23.880933 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9777d542-77d4-4ad3-b2c5-a60b8ec24d68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:49:23.881006 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:23.880985 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9777d542-77d4-4ad3-b2c5-a60b8ec24d68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:49:23.881094 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:23.881055 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9777d542-77d4-4ad3-b2c5-a60b8ec24d68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:49:23.881145 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:23.881113 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9777d542-77d4-4ad3-b2c5-a60b8ec24d68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:49:23.981489 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:23.981465 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9777d542-77d4-4ad3-b2c5-a60b8ec24d68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:49:23.981625 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:23.981499 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9777d542-77d4-4ad3-b2c5-a60b8ec24d68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:49:23.981625 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:23.981523 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kgj5\" (UniqueName: \"kubernetes.io/projected/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-kube-api-access-4kgj5\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9777d542-77d4-4ad3-b2c5-a60b8ec24d68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:49:23.981760 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:23.981642 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9777d542-77d4-4ad3-b2c5-a60b8ec24d68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:49:23.981760 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:23.981678 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9777d542-77d4-4ad3-b2c5-a60b8ec24d68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:49:23.981760 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:23.981704 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9777d542-77d4-4ad3-b2c5-a60b8ec24d68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:49:23.981760 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:23.981753 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9777d542-77d4-4ad3-b2c5-a60b8ec24d68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:49:23.982179 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:23.982148 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9777d542-77d4-4ad3-b2c5-a60b8ec24d68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:49:23.984215 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:23.984188 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9777d542-77d4-4ad3-b2c5-a60b8ec24d68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:49:23.984327 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:23.984196 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9777d542-77d4-4ad3-b2c5-a60b8ec24d68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:49:23.984327 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:23.984297 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9777d542-77d4-4ad3-b2c5-a60b8ec24d68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:49:23.984538 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:23.984524 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9777d542-77d4-4ad3-b2c5-a60b8ec24d68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:49:23.984666 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:23.984643 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9777d542-77d4-4ad3-b2c5-a60b8ec24d68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:49:23.994825 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:23.994801 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kgj5\" (UniqueName: \"kubernetes.io/projected/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-kube-api-access-4kgj5\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9777d542-77d4-4ad3-b2c5-a60b8ec24d68\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:49:24.152443 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.152415 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:49:24.155909 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.155892 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:24.169675 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.169649 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:49:24.176198 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.176171 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:49:24.231886 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.231616 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 31 00:49:24.231886 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.231639 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:49:24.231886 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.231619 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:49:24.231886 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.231698 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-p8r6g\"" Oct 31 00:49:24.231886 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.231699 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 31 00:49:24.231886 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.231738 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 31 00:49:24.231886 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.231769 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 31 00:49:24.231886 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.231778 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 31 00:49:24.231886 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.231859 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 31 00:49:24.284649 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.284624 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:24.284778 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.284658 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:24.284778 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.284688 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:24.284853 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.284809 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqvt4\" (UniqueName: \"kubernetes.io/projected/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-kube-api-access-gqvt4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:24.284853 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.284836 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:24.284916 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.284855 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:24.285031 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.285010 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:24.285101 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.285044 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:24.285101 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.285076 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:24.285187 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.285132 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:24.285187 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.285156 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:24.285285 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.285198 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:24.385646 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.385621 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:24.385817 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.385655 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:24.385817 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.385679 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:24.385817 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.385711 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqvt4\" (UniqueName: \"kubernetes.io/projected/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-kube-api-access-gqvt4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:24.385817 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.385738 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:24.385817 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.385778 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:24.386160 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.385857 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:24.386160 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.385889 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:24.386160 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.385935 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:24.386160 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.385991 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:24.386160 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.386034 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:24.386160 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.386062 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:24.386491 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.386342 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:24.387131 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.387093 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:24.388067 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.388010 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:24.389142 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.389116 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:24.389239 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.389119 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:24.389239 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.389175 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:24.389646 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.389616 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:24.389748 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.389647 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:24.389852 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.389822 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:24.390329 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.390305 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:24.390902 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.390881 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:24.400469 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.400447 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqvt4\" (UniqueName: \"kubernetes.io/projected/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-kube-api-access-gqvt4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:24.458840 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.458816 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:49:24.460616 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:49:24.460596 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9777d542_77d4_4ad3_b2c5_a60b8ec24d68.slice/crio-346f59ca0b528b5d407d011f254d8e9adc952557d0beeccb14e7be712072371a WatchSource:0}: Error finding container 346f59ca0b528b5d407d011f254d8e9adc952557d0beeccb14e7be712072371a: Status 404 returned error can't find the container with id 346f59ca0b528b5d407d011f254d8e9adc952557d0beeccb14e7be712072371a Oct 31 00:49:24.967346 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.967320 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-p2hvj" Oct 31 00:49:24.994338 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.994301 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"9777d542-77d4-4ad3-b2c5-a60b8ec24d68","Type":"ContainerStarted","Data":"34e4ff05d91764d794fc49cfa76ffa6ff6184ae3b9b126a4a724f2e0e49acc32"} Oct 31 00:49:24.994467 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:24.994387 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"9777d542-77d4-4ad3-b2c5-a60b8ec24d68","Type":"ContainerStarted","Data":"346f59ca0b528b5d407d011f254d8e9adc952557d0beeccb14e7be712072371a"} Oct 31 00:49:28.960759 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:28.960732 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-499qg" Oct 31 00:49:29.015521 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:29.015488 5274 generic.go:356] "Generic (PLEG): container finished" podID="9777d542-77d4-4ad3-b2c5-a60b8ec24d68" containerID="34e4ff05d91764d794fc49cfa76ffa6ff6184ae3b9b126a4a724f2e0e49acc32" exitCode=0 Oct 31 00:49:29.015672 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:29.015523 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"9777d542-77d4-4ad3-b2c5-a60b8ec24d68","Type":"ContainerDied","Data":"34e4ff05d91764d794fc49cfa76ffa6ff6184ae3b9b126a4a724f2e0e49acc32"} Oct 31 00:49:30.022777 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:30.022744 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"9777d542-77d4-4ad3-b2c5-a60b8ec24d68","Type":"ContainerStarted","Data":"83ff9e940a17ee7a30376240e8c2c4b7d90e39cbf718bdf619e1322ebde9ab47"} Oct 31 00:49:30.023169 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:30.022806 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"9777d542-77d4-4ad3-b2c5-a60b8ec24d68","Type":"ContainerStarted","Data":"76180109ab66e857297d5108073731e23f4f2fe16cce995d736a8bd191c0c61b"} Oct 31 00:49:30.023169 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:30.022998 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:49:30.051684 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:30.051628 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=3.7653028060000002 podStartE2EDuration="7.051614397s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.286311591s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:30.05069493 +0000 UTC m=+3006.687513646" watchObservedRunningTime="2026-10-31 00:49:30.051614397 +0000 UTC m=+3006.688433114" Oct 31 00:49:33.961192 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:33.961161 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:33.968403 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:33.968378 5274 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:34.062071 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:34.062049 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:34.062198 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:34.062158 5274 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:34.066354 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:34.066337 5274 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") DevicePath \"csi-3beb91b3ebf177571ca474589da3560cd18787cfa50b2a0b9bfef85661e6cf21\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:34.156902 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:34.156872 5274 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/534eb232558c8505fa55558aa87cc96e2d2b0adb946edb717939ec387be54e97/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:34.173931 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:34.173910 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:34.381355 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:34.381332 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:34.668561 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:34.668535 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:49:34.670673 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:49:34.670649 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf42a5beb_ad0f_43fc_97ad_0af6febd68ee.slice/crio-8d3deeac9c8c68e4df134436e2c9e5a7d156a1a1351124d9f978139f3b6593fc WatchSource:0}: Error finding container 8d3deeac9c8c68e4df134436e2c9e5a7d156a1a1351124d9f978139f3b6593fc: Status 404 returned error can't find the container with id 8d3deeac9c8c68e4df134436e2c9e5a7d156a1a1351124d9f978139f3b6593fc Oct 31 00:49:35.047816 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:35.047728 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"f42a5beb-ad0f-43fc-97ad-0af6febd68ee","Type":"ContainerStarted","Data":"8d3deeac9c8c68e4df134436e2c9e5a7d156a1a1351124d9f978139f3b6593fc"} Oct 31 00:49:36.029440 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:36.029410 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:49:36.055224 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:36.055180 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"f42a5beb-ad0f-43fc-97ad-0af6febd68ee","Type":"ContainerStarted","Data":"b749b49dad5719aff95215bfcc3425c768d9580ba426f0558c636d3ca8cb67b4"} Oct 31 00:49:39.071582 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:39.071548 5274 generic.go:356] "Generic (PLEG): container finished" podID="f42a5beb-ad0f-43fc-97ad-0af6febd68ee" containerID="b749b49dad5719aff95215bfcc3425c768d9580ba426f0558c636d3ca8cb67b4" exitCode=0 Oct 31 00:49:39.072016 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:39.071621 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"f42a5beb-ad0f-43fc-97ad-0af6febd68ee","Type":"ContainerDied","Data":"b749b49dad5719aff95215bfcc3425c768d9580ba426f0558c636d3ca8cb67b4"} Oct 31 00:49:40.079822 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:40.079788 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"f42a5beb-ad0f-43fc-97ad-0af6febd68ee","Type":"ContainerStarted","Data":"668c602e5308c4841913ecdfb7c4f1ba3c6b8fb5db6f39a0cb38f9087339d140"} Oct 31 00:49:40.080320 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:40.079864 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"f42a5beb-ad0f-43fc-97ad-0af6febd68ee","Type":"ContainerStarted","Data":"373b52de232d2413f8ca9ed9494345c3a19b7f6f340b7f3e16aa5d0823d5c703"} Oct 31 00:49:40.080320 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:40.079902 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"f42a5beb-ad0f-43fc-97ad-0af6febd68ee","Type":"ContainerStarted","Data":"7be120732babac11b637dcb826688d00ea943679e124ffd2358545eef3d76385"} Oct 31 00:49:40.116170 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:40.116124 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=12.630827815 podStartE2EDuration="16.116110757s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.485282942s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:40.112384304 +0000 UTC m=+3016.749203018" watchObservedRunningTime="2026-10-31 00:49:40.116110757 +0000 UTC m=+3016.752929471" Oct 31 00:49:44.381406 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:44.381380 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:47.732165 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:47.732118 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:49:47.732650 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:47.732341 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="6b16f028-c18a-4c4d-9a88-d6c961bfa589" containerName="otc-container" containerID="cri-o://001b2d9b6da75c3217fca9993f39291f45c0bdcdddc7fb091bc7beefaa92ae7c" gracePeriod=30 Oct 31 00:49:48.987807 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:48.987780 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:49.092282 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.092207 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b16f028-c18a-4c4d-9a88-d6c961bfa589-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6b16f028-c18a-4c4d-9a88-d6c961bfa589-otlp-tls-certs\") pod \"6b16f028-c18a-4c4d-9a88-d6c961bfa589\" (UID: \"6b16f028-c18a-4c4d-9a88-d6c961bfa589\") " Oct 31 00:49:49.092282 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.092240 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b16f028-c18a-4c4d-9a88-d6c961bfa589-otc-internal\" (UniqueName: \"kubernetes.io/configmap/6b16f028-c18a-4c4d-9a88-d6c961bfa589-otc-internal\") pod \"6b16f028-c18a-4c4d-9a88-d6c961bfa589\" (UID: \"6b16f028-c18a-4c4d-9a88-d6c961bfa589\") " Oct 31 00:49:49.092469 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.092297 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b16f028-c18a-4c4d-9a88-d6c961bfa589-tls-certs\" (UniqueName: \"kubernetes.io/secret/6b16f028-c18a-4c4d-9a88-d6c961bfa589-tls-certs\") pod \"6b16f028-c18a-4c4d-9a88-d6c961bfa589\" (UID: \"6b16f028-c18a-4c4d-9a88-d6c961bfa589\") " Oct 31 00:49:49.092469 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.092325 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b16f028-c18a-4c4d-9a88-d6c961bfa589-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6b16f028-c18a-4c4d-9a88-d6c961bfa589-monitoring-tls-certs\") pod \"6b16f028-c18a-4c4d-9a88-d6c961bfa589\" (UID: \"6b16f028-c18a-4c4d-9a88-d6c961bfa589\") " Oct 31 00:49:49.092469 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.092362 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b16f028-c18a-4c4d-9a88-d6c961bfa589-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6b16f028-c18a-4c4d-9a88-d6c961bfa589-data-science-collector-ta-client-cert\") pod \"6b16f028-c18a-4c4d-9a88-d6c961bfa589\" (UID: \"6b16f028-c18a-4c4d-9a88-d6c961bfa589\") " Oct 31 00:49:49.092469 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.092392 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b16f028-c18a-4c4d-9a88-d6c961bfa589-kube-api-access-b9pz6\" (UniqueName: \"kubernetes.io/projected/6b16f028-c18a-4c4d-9a88-d6c961bfa589-kube-api-access-b9pz6\") pod \"6b16f028-c18a-4c4d-9a88-d6c961bfa589\" (UID: \"6b16f028-c18a-4c4d-9a88-d6c961bfa589\") " Oct 31 00:49:49.092692 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.092658 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b16f028-c18a-4c4d-9a88-d6c961bfa589-otc-internal" pod "6b16f028-c18a-4c4d-9a88-d6c961bfa589" (UID: "6b16f028-c18a-4c4d-9a88-d6c961bfa589"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:49.094723 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.094693 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b16f028-c18a-4c4d-9a88-d6c961bfa589-otlp-tls-certs" pod "6b16f028-c18a-4c4d-9a88-d6c961bfa589" (UID: "6b16f028-c18a-4c4d-9a88-d6c961bfa589"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:49.094840 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.094730 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b16f028-c18a-4c4d-9a88-d6c961bfa589-kube-api-access-b9pz6" pod "6b16f028-c18a-4c4d-9a88-d6c961bfa589" (UID: "6b16f028-c18a-4c4d-9a88-d6c961bfa589"). InnerVolumeSpecName "kube-api-access-b9pz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:49.094840 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.094826 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b16f028-c18a-4c4d-9a88-d6c961bfa589-monitoring-tls-certs" pod "6b16f028-c18a-4c4d-9a88-d6c961bfa589" (UID: "6b16f028-c18a-4c4d-9a88-d6c961bfa589"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:49.094974 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.094884 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b16f028-c18a-4c4d-9a88-d6c961bfa589-data-science-collector-ta-client-cert" pod "6b16f028-c18a-4c4d-9a88-d6c961bfa589" (UID: "6b16f028-c18a-4c4d-9a88-d6c961bfa589"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:49.094974 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.094916 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b16f028-c18a-4c4d-9a88-d6c961bfa589-tls-certs" pod "6b16f028-c18a-4c4d-9a88-d6c961bfa589" (UID: "6b16f028-c18a-4c4d-9a88-d6c961bfa589"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:49.125981 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.125927 5274 generic.go:356] "Generic (PLEG): container finished" podID="6b16f028-c18a-4c4d-9a88-d6c961bfa589" containerID="001b2d9b6da75c3217fca9993f39291f45c0bdcdddc7fb091bc7beefaa92ae7c" exitCode=0 Oct 31 00:49:49.126108 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.125986 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"6b16f028-c18a-4c4d-9a88-d6c961bfa589","Type":"ContainerDied","Data":"001b2d9b6da75c3217fca9993f39291f45c0bdcdddc7fb091bc7beefaa92ae7c"} Oct 31 00:49:49.126108 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.126052 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"6b16f028-c18a-4c4d-9a88-d6c961bfa589","Type":"ContainerDied","Data":"29f5553c23725386d94aaa8a8ba09c29f1b1634edd02ce4ba058456e0abb83e5"} Oct 31 00:49:49.126108 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.126074 5274 scope.go:122] "RemoveContainer" containerID="001b2d9b6da75c3217fca9993f39291f45c0bdcdddc7fb091bc7beefaa92ae7c" Oct 31 00:49:49.126108 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.125994 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:49.135451 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.135435 5274 scope.go:122] "RemoveContainer" containerID="001b2d9b6da75c3217fca9993f39291f45c0bdcdddc7fb091bc7beefaa92ae7c" Oct 31 00:49:49.135747 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:49:49.135729 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"001b2d9b6da75c3217fca9993f39291f45c0bdcdddc7fb091bc7beefaa92ae7c\": container with ID starting with 001b2d9b6da75c3217fca9993f39291f45c0bdcdddc7fb091bc7beefaa92ae7c not found: ID does not exist" containerID="001b2d9b6da75c3217fca9993f39291f45c0bdcdddc7fb091bc7beefaa92ae7c" Oct 31 00:49:49.135785 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.135755 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"001b2d9b6da75c3217fca9993f39291f45c0bdcdddc7fb091bc7beefaa92ae7c"} err="failed to get container status \"001b2d9b6da75c3217fca9993f39291f45c0bdcdddc7fb091bc7beefaa92ae7c\": rpc error: code = NotFound desc = could not find container \"001b2d9b6da75c3217fca9993f39291f45c0bdcdddc7fb091bc7beefaa92ae7c\": container with ID starting with 001b2d9b6da75c3217fca9993f39291f45c0bdcdddc7fb091bc7beefaa92ae7c not found: ID does not exist" Oct 31 00:49:49.150799 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.150774 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:49:49.155532 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.155511 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:49:49.183358 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.183339 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:49:49.183712 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.183699 5274 cpu_manager.go:414] "containerMap: removing container" podUID="6b16f028-c18a-4c4d-9a88-d6c961bfa589" containerName="otc-container" Oct 31 00:49:49.183712 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.183713 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b16f028-c18a-4c4d-9a88-d6c961bfa589" containerName="otc-container" Oct 31 00:49:49.183854 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.183793 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b16f028-c18a-4c4d-9a88-d6c961bfa589" containerName="otc-container" Oct 31 00:49:49.187892 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.187877 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:49.192091 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.192068 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:49:49.192228 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.192114 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:49:49.192228 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.192145 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-cp86t\"" Oct 31 00:49:49.192678 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.192666 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:49:49.192754 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.192700 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:49:49.192754 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.192720 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:49:49.193045 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.193028 5274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b16f028-c18a-4c4d-9a88-d6c961bfa589-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:49.193159 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.193046 5274 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6b16f028-c18a-4c4d-9a88-d6c961bfa589-monitoring-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:49.193159 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.193059 5274 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6b16f028-c18a-4c4d-9a88-d6c961bfa589-data-science-collector-ta-client-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:49.193159 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.193069 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9pz6\" (UniqueName: \"kubernetes.io/projected/6b16f028-c18a-4c4d-9a88-d6c961bfa589-kube-api-access-b9pz6\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:49.193159 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.193078 5274 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6b16f028-c18a-4c4d-9a88-d6c961bfa589-otlp-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:49.193159 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.193086 5274 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6b16f028-c18a-4c4d-9a88-d6c961bfa589-otc-internal\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:49.198630 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.198612 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:49:49.294323 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.294301 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc65w\" (UniqueName: \"kubernetes.io/projected/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-kube-api-access-zc65w\") pod \"data-science-collector-collector-0\" (UID: \"d3fda09c-8670-4f6c-90c2-f557ed1dd2a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:49.294461 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.294342 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"d3fda09c-8670-4f6c-90c2-f557ed1dd2a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:49.294461 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.294365 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d3fda09c-8670-4f6c-90c2-f557ed1dd2a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:49.294547 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.294482 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d3fda09c-8670-4f6c-90c2-f557ed1dd2a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:49.294547 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.294516 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d3fda09c-8670-4f6c-90c2-f557ed1dd2a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:49.294547 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.294539 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"d3fda09c-8670-4f6c-90c2-f557ed1dd2a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:49.382188 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.382167 5274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:49.387352 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.387334 5274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:49.395525 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.395507 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d3fda09c-8670-4f6c-90c2-f557ed1dd2a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:49.395606 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.395541 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d3fda09c-8670-4f6c-90c2-f557ed1dd2a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:49.395606 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.395562 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"d3fda09c-8670-4f6c-90c2-f557ed1dd2a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:49.395606 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.395598 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc65w\" (UniqueName: \"kubernetes.io/projected/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-kube-api-access-zc65w\") pod \"data-science-collector-collector-0\" (UID: \"d3fda09c-8670-4f6c-90c2-f557ed1dd2a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:49.395744 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.395629 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"d3fda09c-8670-4f6c-90c2-f557ed1dd2a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:49.395744 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.395650 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d3fda09c-8670-4f6c-90c2-f557ed1dd2a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:49.396411 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.396386 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"d3fda09c-8670-4f6c-90c2-f557ed1dd2a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:49.398086 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.398064 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d3fda09c-8670-4f6c-90c2-f557ed1dd2a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:49.398157 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.398120 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d3fda09c-8670-4f6c-90c2-f557ed1dd2a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:49.398157 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.398148 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d3fda09c-8670-4f6c-90c2-f557ed1dd2a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:49.398249 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.398234 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"d3fda09c-8670-4f6c-90c2-f557ed1dd2a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:49.410455 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.410432 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc65w\" (UniqueName: \"kubernetes.io/projected/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-kube-api-access-zc65w\") pod \"data-science-collector-collector-0\" (UID: \"d3fda09c-8670-4f6c-90c2-f557ed1dd2a4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:49.480835 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.480812 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b16f028-c18a-4c4d-9a88-d6c961bfa589" path="/var/lib/kubelet/pods/6b16f028-c18a-4c4d-9a88-d6c961bfa589/volumes" Oct 31 00:49:49.500556 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.500538 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:49:49.774475 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.774451 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:49:49.776379 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:49:49.776352 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3fda09c_8670_4f6c_90c2_f557ed1dd2a4.slice/crio-a5e45881f861a7cde4d3596ae9ec9b9ff6f37750b38bea9fab066fded4c50c14 WatchSource:0}: Error finding container a5e45881f861a7cde4d3596ae9ec9b9ff6f37750b38bea9fab066fded4c50c14: Status 404 returned error can't find the container with id a5e45881f861a7cde4d3596ae9ec9b9ff6f37750b38bea9fab066fded4c50c14 Oct 31 00:49:49.778543 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.778515 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:49:49.778602 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:49.778569 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:49:50.133542 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:50.133499 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d3fda09c-8670-4f6c-90c2-f557ed1dd2a4","Type":"ContainerStarted","Data":"2537060b548bcce6dad759768327019f2ff390bd7bd61bbb9b53194b6c0662f2"} Oct 31 00:49:50.134052 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:50.133565 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d3fda09c-8670-4f6c-90c2-f557ed1dd2a4","Type":"ContainerStarted","Data":"a5e45881f861a7cde4d3596ae9ec9b9ff6f37750b38bea9fab066fded4c50c14"} Oct 31 00:49:50.139374 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:50.139354 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:49:50.155401 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:49:50.155339 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.155324216 podStartE2EDuration="1.155324216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:50.154162836 +0000 UTC m=+3026.790981552" watchObservedRunningTime="2026-10-31 00:49:50.155324216 +0000 UTC m=+3026.792142932" Oct 31 00:49:50.899276 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:49:50.899245 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:50:07.824916 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:07.824878 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:50:07.825539 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:07.825134 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="d3fda09c-8670-4f6c-90c2-f557ed1dd2a4" containerName="otc-container" containerID="cri-o://2537060b548bcce6dad759768327019f2ff390bd7bd61bbb9b53194b6c0662f2" gracePeriod=30 Oct 31 00:50:09.069563 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.069540 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:09.153440 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.153371 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-otc-internal\" (UniqueName: \"kubernetes.io/configmap/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-otc-internal\") pod \"d3fda09c-8670-4f6c-90c2-f557ed1dd2a4\" (UID: \"d3fda09c-8670-4f6c-90c2-f557ed1dd2a4\") " Oct 31 00:50:09.153440 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.153407 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-kube-api-access-zc65w\" (UniqueName: \"kubernetes.io/projected/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-kube-api-access-zc65w\") pod \"d3fda09c-8670-4f6c-90c2-f557ed1dd2a4\" (UID: \"d3fda09c-8670-4f6c-90c2-f557ed1dd2a4\") " Oct 31 00:50:09.153655 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.153493 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-otlp-tls-certs\") pod \"d3fda09c-8670-4f6c-90c2-f557ed1dd2a4\" (UID: \"d3fda09c-8670-4f6c-90c2-f557ed1dd2a4\") " Oct 31 00:50:09.153655 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.153519 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-tls-certs\") pod \"d3fda09c-8670-4f6c-90c2-f557ed1dd2a4\" (UID: \"d3fda09c-8670-4f6c-90c2-f557ed1dd2a4\") " Oct 31 00:50:09.153655 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.153566 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-data-science-collector-ta-client-cert\") pod \"d3fda09c-8670-4f6c-90c2-f557ed1dd2a4\" (UID: \"d3fda09c-8670-4f6c-90c2-f557ed1dd2a4\") " Oct 31 00:50:09.153655 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.153591 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-monitoring-tls-certs\") pod \"d3fda09c-8670-4f6c-90c2-f557ed1dd2a4\" (UID: \"d3fda09c-8670-4f6c-90c2-f557ed1dd2a4\") " Oct 31 00:50:09.153864 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.153827 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-otc-internal" pod "d3fda09c-8670-4f6c-90c2-f557ed1dd2a4" (UID: "d3fda09c-8670-4f6c-90c2-f557ed1dd2a4"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:09.155865 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.155834 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-tls-certs" pod "d3fda09c-8670-4f6c-90c2-f557ed1dd2a4" (UID: "d3fda09c-8670-4f6c-90c2-f557ed1dd2a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:09.155865 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.155856 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-data-science-collector-ta-client-cert" pod "d3fda09c-8670-4f6c-90c2-f557ed1dd2a4" (UID: "d3fda09c-8670-4f6c-90c2-f557ed1dd2a4"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:09.156089 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.155869 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-otlp-tls-certs" pod "d3fda09c-8670-4f6c-90c2-f557ed1dd2a4" (UID: "d3fda09c-8670-4f6c-90c2-f557ed1dd2a4"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:09.156135 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.156116 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-kube-api-access-zc65w" pod "d3fda09c-8670-4f6c-90c2-f557ed1dd2a4" (UID: "d3fda09c-8670-4f6c-90c2-f557ed1dd2a4"). InnerVolumeSpecName "kube-api-access-zc65w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:09.156176 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.156138 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-monitoring-tls-certs" pod "d3fda09c-8670-4f6c-90c2-f557ed1dd2a4" (UID: "d3fda09c-8670-4f6c-90c2-f557ed1dd2a4"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:09.234938 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.234909 5274 generic.go:356] "Generic (PLEG): container finished" podID="d3fda09c-8670-4f6c-90c2-f557ed1dd2a4" containerID="2537060b548bcce6dad759768327019f2ff390bd7bd61bbb9b53194b6c0662f2" exitCode=0 Oct 31 00:50:09.235070 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.234968 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:09.235070 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.234998 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d3fda09c-8670-4f6c-90c2-f557ed1dd2a4","Type":"ContainerDied","Data":"2537060b548bcce6dad759768327019f2ff390bd7bd61bbb9b53194b6c0662f2"} Oct 31 00:50:09.235070 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.235055 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d3fda09c-8670-4f6c-90c2-f557ed1dd2a4","Type":"ContainerDied","Data":"a5e45881f861a7cde4d3596ae9ec9b9ff6f37750b38bea9fab066fded4c50c14"} Oct 31 00:50:09.235235 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.235087 5274 scope.go:122] "RemoveContainer" containerID="2537060b548bcce6dad759768327019f2ff390bd7bd61bbb9b53194b6c0662f2" Oct 31 00:50:09.245213 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.245196 5274 scope.go:122] "RemoveContainer" containerID="2537060b548bcce6dad759768327019f2ff390bd7bd61bbb9b53194b6c0662f2" Oct 31 00:50:09.245459 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:50:09.245440 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2537060b548bcce6dad759768327019f2ff390bd7bd61bbb9b53194b6c0662f2\": container with ID starting with 2537060b548bcce6dad759768327019f2ff390bd7bd61bbb9b53194b6c0662f2 not found: ID does not exist" containerID="2537060b548bcce6dad759768327019f2ff390bd7bd61bbb9b53194b6c0662f2" Oct 31 00:50:09.245505 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.245467 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2537060b548bcce6dad759768327019f2ff390bd7bd61bbb9b53194b6c0662f2"} err="failed to get container status \"2537060b548bcce6dad759768327019f2ff390bd7bd61bbb9b53194b6c0662f2\": rpc error: code = NotFound desc = could not find container \"2537060b548bcce6dad759768327019f2ff390bd7bd61bbb9b53194b6c0662f2\": container with ID starting with 2537060b548bcce6dad759768327019f2ff390bd7bd61bbb9b53194b6c0662f2 not found: ID does not exist" Oct 31 00:50:09.254179 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.254156 5274 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-otlp-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:09.254179 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.254177 5274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:09.254308 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.254187 5274 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-data-science-collector-ta-client-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:09.254308 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.254197 5274 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-monitoring-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:09.254308 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.254206 5274 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-otc-internal\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:09.254308 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.254215 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc65w\" (UniqueName: \"kubernetes.io/projected/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4-kube-api-access-zc65w\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:09.259339 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.259313 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:50:09.264528 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.264511 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:50:09.286090 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.286067 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-p2hvj"] Oct 31 00:50:09.286253 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.286233 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-p2hvj" podUID="149823e9-b0e4-476c-9d37-d4581e7ea7d1" containerName="kube-rbac-proxy" containerID="cri-o://edb61927ba6edbbfebed49df327d5d7d7aaa15c5dc32ffedb9258645f93e5db8" gracePeriod=30 Oct 31 00:50:09.298630 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.298607 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:50:09.299886 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.299869 5274 cpu_manager.go:414] "containerMap: removing container" podUID="d3fda09c-8670-4f6c-90c2-f557ed1dd2a4" containerName="otc-container" Oct 31 00:50:09.300058 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.300029 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3fda09c-8670-4f6c-90c2-f557ed1dd2a4" containerName="otc-container" Oct 31 00:50:09.300443 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.300415 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3fda09c-8670-4f6c-90c2-f557ed1dd2a4" containerName="otc-container" Oct 31 00:50:09.307359 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.307338 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:09.311267 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.311249 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-499qg"] Oct 31 00:50:09.311418 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.311399 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-499qg" podUID="58d39fb2-6f91-4f59-aa7c-45e69e43cb12" containerName="korrel8r" containerID="cri-o://fb76b61862f6ca9f45a389d10cfe5da76ea8b6356f3c4e58b5c93118b7e1298d" gracePeriod=30 Oct 31 00:50:09.314122 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.313591 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:50:09.314122 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.313697 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:50:09.314122 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.313806 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:50:09.314122 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.313886 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-cp86t\"" Oct 31 00:50:09.317023 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.316768 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:50:09.317023 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.316854 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:50:09.319105 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.319083 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:50:09.355646 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.355600 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/976644b4-136a-42bd-95c8-c5dd1e415ac6-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"976644b4-136a-42bd-95c8-c5dd1e415ac6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:09.356153 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.355678 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vpkz\" (UniqueName: \"kubernetes.io/projected/976644b4-136a-42bd-95c8-c5dd1e415ac6-kube-api-access-8vpkz\") pod \"data-science-collector-collector-0\" (UID: \"976644b4-136a-42bd-95c8-c5dd1e415ac6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:09.356153 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.355775 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/976644b4-136a-42bd-95c8-c5dd1e415ac6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"976644b4-136a-42bd-95c8-c5dd1e415ac6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:09.356153 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.355825 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/976644b4-136a-42bd-95c8-c5dd1e415ac6-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"976644b4-136a-42bd-95c8-c5dd1e415ac6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:09.356153 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.355942 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/976644b4-136a-42bd-95c8-c5dd1e415ac6-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"976644b4-136a-42bd-95c8-c5dd1e415ac6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:09.356153 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.355981 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/976644b4-136a-42bd-95c8-c5dd1e415ac6-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"976644b4-136a-42bd-95c8-c5dd1e415ac6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:09.457630 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.457600 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/976644b4-136a-42bd-95c8-c5dd1e415ac6-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"976644b4-136a-42bd-95c8-c5dd1e415ac6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:09.457798 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.457646 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vpkz\" (UniqueName: \"kubernetes.io/projected/976644b4-136a-42bd-95c8-c5dd1e415ac6-kube-api-access-8vpkz\") pod \"data-science-collector-collector-0\" (UID: \"976644b4-136a-42bd-95c8-c5dd1e415ac6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:09.457798 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.457694 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/976644b4-136a-42bd-95c8-c5dd1e415ac6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"976644b4-136a-42bd-95c8-c5dd1e415ac6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:09.457798 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.457724 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/976644b4-136a-42bd-95c8-c5dd1e415ac6-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"976644b4-136a-42bd-95c8-c5dd1e415ac6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:09.458000 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.457814 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/976644b4-136a-42bd-95c8-c5dd1e415ac6-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"976644b4-136a-42bd-95c8-c5dd1e415ac6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:09.458000 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.457841 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/976644b4-136a-42bd-95c8-c5dd1e415ac6-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"976644b4-136a-42bd-95c8-c5dd1e415ac6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:09.464478 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.464422 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/976644b4-136a-42bd-95c8-c5dd1e415ac6-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"976644b4-136a-42bd-95c8-c5dd1e415ac6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:09.466554 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.466498 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/976644b4-136a-42bd-95c8-c5dd1e415ac6-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"976644b4-136a-42bd-95c8-c5dd1e415ac6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:09.467484 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.467446 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/976644b4-136a-42bd-95c8-c5dd1e415ac6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"976644b4-136a-42bd-95c8-c5dd1e415ac6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:09.468155 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.468132 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/976644b4-136a-42bd-95c8-c5dd1e415ac6-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"976644b4-136a-42bd-95c8-c5dd1e415ac6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:09.473907 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.473882 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/976644b4-136a-42bd-95c8-c5dd1e415ac6-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"976644b4-136a-42bd-95c8-c5dd1e415ac6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:09.478567 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.478520 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vpkz\" (UniqueName: \"kubernetes.io/projected/976644b4-136a-42bd-95c8-c5dd1e415ac6-kube-api-access-8vpkz\") pod \"data-science-collector-collector-0\" (UID: \"976644b4-136a-42bd-95c8-c5dd1e415ac6\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:09.485324 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.484808 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3fda09c-8670-4f6c-90c2-f557ed1dd2a4" path="/var/lib/kubelet/pods/d3fda09c-8670-4f6c-90c2-f557ed1dd2a4/volumes" Oct 31 00:50:09.521665 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.521638 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:50:09.521972 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.521926 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="f42a5beb-ad0f-43fc-97ad-0af6febd68ee" containerName="thanos-sidecar" containerID="cri-o://668c602e5308c4841913ecdfb7c4f1ba3c6b8fb5db6f39a0cb38f9087339d140" gracePeriod=600 Oct 31 00:50:09.522130 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.521937 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="f42a5beb-ad0f-43fc-97ad-0af6febd68ee" containerName="config-reloader" containerID="cri-o://373b52de232d2413f8ca9ed9494345c3a19b7f6f340b7f3e16aa5d0823d5c703" gracePeriod=600 Oct 31 00:50:09.522197 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.521918 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="f42a5beb-ad0f-43fc-97ad-0af6febd68ee" containerName="prometheus" containerID="cri-o://7be120732babac11b637dcb826688d00ea943679e124ffd2358545eef3d76385" gracePeriod=600 Oct 31 00:50:09.535752 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.535724 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:50:09.536038 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.536015 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="9777d542-77d4-4ad3-b2c5-a60b8ec24d68" containerName="alertmanager" containerID="cri-o://76180109ab66e857297d5108073731e23f4f2fe16cce995d736a8bd191c0c61b" gracePeriod=120 Oct 31 00:50:09.536143 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.536113 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="9777d542-77d4-4ad3-b2c5-a60b8ec24d68" containerName="config-reloader" containerID="cri-o://83ff9e940a17ee7a30376240e8c2c4b7d90e39cbf718bdf619e1322ebde9ab47" gracePeriod=120 Oct 31 00:50:09.656491 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.656466 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-p2hvj" Oct 31 00:50:09.660215 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.660162 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-499qg" Oct 31 00:50:09.663046 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.663024 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:50:09.663262 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.663251 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:09.761993 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.761251 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/149823e9-b0e4-476c-9d37-d4581e7ea7d1-tls\" (UniqueName: \"kubernetes.io/secret/149823e9-b0e4-476c-9d37-d4581e7ea7d1-tls\") pod \"149823e9-b0e4-476c-9d37-d4581e7ea7d1\" (UID: \"149823e9-b0e4-476c-9d37-d4581e7ea7d1\") " Oct 31 00:50:09.761993 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.761314 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58d39fb2-6f91-4f59-aa7c-45e69e43cb12-tls\" (UniqueName: \"kubernetes.io/secret/58d39fb2-6f91-4f59-aa7c-45e69e43cb12-tls\") pod \"58d39fb2-6f91-4f59-aa7c-45e69e43cb12\" (UID: \"58d39fb2-6f91-4f59-aa7c-45e69e43cb12\") " Oct 31 00:50:09.761993 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.761355 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/149823e9-b0e4-476c-9d37-d4581e7ea7d1-kube-api-access-cct76\" (UniqueName: \"kubernetes.io/projected/149823e9-b0e4-476c-9d37-d4581e7ea7d1-kube-api-access-cct76\") pod \"149823e9-b0e4-476c-9d37-d4581e7ea7d1\" (UID: \"149823e9-b0e4-476c-9d37-d4581e7ea7d1\") " Oct 31 00:50:09.761993 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.761405 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/149823e9-b0e4-476c-9d37-d4581e7ea7d1-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/149823e9-b0e4-476c-9d37-d4581e7ea7d1-prometheus-client-cert\") pod \"149823e9-b0e4-476c-9d37-d4581e7ea7d1\" (UID: \"149823e9-b0e4-476c-9d37-d4581e7ea7d1\") " Oct 31 00:50:09.761993 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.761444 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58d39fb2-6f91-4f59-aa7c-45e69e43cb12-kube-api-access-67m5x\" (UniqueName: \"kubernetes.io/projected/58d39fb2-6f91-4f59-aa7c-45e69e43cb12-kube-api-access-67m5x\") pod \"58d39fb2-6f91-4f59-aa7c-45e69e43cb12\" (UID: \"58d39fb2-6f91-4f59-aa7c-45e69e43cb12\") " Oct 31 00:50:09.761993 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.761519 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/149823e9-b0e4-476c-9d37-d4581e7ea7d1-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/149823e9-b0e4-476c-9d37-d4581e7ea7d1-prometheus-ca\") pod \"149823e9-b0e4-476c-9d37-d4581e7ea7d1\" (UID: \"149823e9-b0e4-476c-9d37-d4581e7ea7d1\") " Oct 31 00:50:09.761993 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.761559 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/149823e9-b0e4-476c-9d37-d4581e7ea7d1-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/149823e9-b0e4-476c-9d37-d4581e7ea7d1-kube-rbac-proxy-config\") pod \"149823e9-b0e4-476c-9d37-d4581e7ea7d1\" (UID: \"149823e9-b0e4-476c-9d37-d4581e7ea7d1\") " Oct 31 00:50:09.761993 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.761609 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58d39fb2-6f91-4f59-aa7c-45e69e43cb12-config\" (UniqueName: \"kubernetes.io/configmap/58d39fb2-6f91-4f59-aa7c-45e69e43cb12-config\") pod \"58d39fb2-6f91-4f59-aa7c-45e69e43cb12\" (UID: \"58d39fb2-6f91-4f59-aa7c-45e69e43cb12\") " Oct 31 00:50:09.762578 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.762474 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58d39fb2-6f91-4f59-aa7c-45e69e43cb12-config" pod "58d39fb2-6f91-4f59-aa7c-45e69e43cb12" (UID: "58d39fb2-6f91-4f59-aa7c-45e69e43cb12"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:09.763505 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.763474 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/149823e9-b0e4-476c-9d37-d4581e7ea7d1-prometheus-ca" pod "149823e9-b0e4-476c-9d37-d4581e7ea7d1" (UID: "149823e9-b0e4-476c-9d37-d4581e7ea7d1"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:09.793919 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.793811 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/149823e9-b0e4-476c-9d37-d4581e7ea7d1-tls" pod "149823e9-b0e4-476c-9d37-d4581e7ea7d1" (UID: "149823e9-b0e4-476c-9d37-d4581e7ea7d1"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:09.794163 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.794054 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58d39fb2-6f91-4f59-aa7c-45e69e43cb12-tls" pod "58d39fb2-6f91-4f59-aa7c-45e69e43cb12" (UID: "58d39fb2-6f91-4f59-aa7c-45e69e43cb12"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:09.794163 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.794062 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d39fb2-6f91-4f59-aa7c-45e69e43cb12-kube-api-access-67m5x" pod "58d39fb2-6f91-4f59-aa7c-45e69e43cb12" (UID: "58d39fb2-6f91-4f59-aa7c-45e69e43cb12"). InnerVolumeSpecName "kube-api-access-67m5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:09.795367 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.795341 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/149823e9-b0e4-476c-9d37-d4581e7ea7d1-prometheus-client-cert" pod "149823e9-b0e4-476c-9d37-d4581e7ea7d1" (UID: "149823e9-b0e4-476c-9d37-d4581e7ea7d1"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:09.796988 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.796938 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/149823e9-b0e4-476c-9d37-d4581e7ea7d1-kube-rbac-proxy-config" pod "149823e9-b0e4-476c-9d37-d4581e7ea7d1" (UID: "149823e9-b0e4-476c-9d37-d4581e7ea7d1"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:09.799108 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.799071 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/149823e9-b0e4-476c-9d37-d4581e7ea7d1-kube-api-access-cct76" pod "149823e9-b0e4-476c-9d37-d4581e7ea7d1" (UID: "149823e9-b0e4-476c-9d37-d4581e7ea7d1"). InnerVolumeSpecName "kube-api-access-cct76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:09.863284 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.863205 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/58d39fb2-6f91-4f59-aa7c-45e69e43cb12-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:09.863284 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.863244 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/149823e9-b0e4-476c-9d37-d4581e7ea7d1-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:09.863284 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.863251 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/58d39fb2-6f91-4f59-aa7c-45e69e43cb12-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:09.863284 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.863259 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cct76\" (UniqueName: \"kubernetes.io/projected/149823e9-b0e4-476c-9d37-d4581e7ea7d1-kube-api-access-cct76\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:09.863284 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.863269 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/149823e9-b0e4-476c-9d37-d4581e7ea7d1-prometheus-client-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:09.863284 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.863277 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67m5x\" (UniqueName: \"kubernetes.io/projected/58d39fb2-6f91-4f59-aa7c-45e69e43cb12-kube-api-access-67m5x\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:09.863284 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.863285 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/149823e9-b0e4-476c-9d37-d4581e7ea7d1-prometheus-ca\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:09.863284 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:09.863294 5274 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/149823e9-b0e4-476c-9d37-d4581e7ea7d1-kube-rbac-proxy-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.035413 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.033713 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:50:10.035413 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:50:10.034528 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod976644b4_136a_42bd_95c8_c5dd1e415ac6.slice/crio-a2063e4cdf4b6b748cfdd16d1e13099ec8dcd01c5b5f326cf081a5e0670593a8 WatchSource:0}: Error finding container a2063e4cdf4b6b748cfdd16d1e13099ec8dcd01c5b5f326cf081a5e0670593a8: Status 404 returned error can't find the container with id a2063e4cdf4b6b748cfdd16d1e13099ec8dcd01c5b5f326cf081a5e0670593a8 Oct 31 00:50:10.037386 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.037346 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:50:10.037498 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.037424 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:50:10.100271 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.100240 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:10.166147 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.166121 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " Oct 31 00:50:10.166264 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.166253 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " Oct 31 00:50:10.166316 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.166289 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-config\" (UniqueName: \"kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-config\") pod \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " Oct 31 00:50:10.166382 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.166316 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-tls-assets\" (UniqueName: \"kubernetes.io/projected/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-tls-assets\") pod \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " Oct 31 00:50:10.166382 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.166372 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-config-out\" (UniqueName: \"kubernetes.io/empty-dir/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-config-out\") pod \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " Oct 31 00:50:10.166495 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.166437 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-web-config\" (UniqueName: \"kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-web-config\") pod \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " Oct 31 00:50:10.166495 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.166483 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " Oct 31 00:50:10.166583 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.166537 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " Oct 31 00:50:10.166583 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.166572 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-kube-api-access-gqvt4\" (UniqueName: \"kubernetes.io/projected/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-kube-api-access-gqvt4\") pod \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " Oct 31 00:50:10.166695 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.166604 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " Oct 31 00:50:10.166695 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.166632 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-thanos-prometheus-http-client-file\") pod \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " Oct 31 00:50:10.166695 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.166672 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " Oct 31 00:50:10.166869 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.166704 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-secret-prometheus-web-tls-ca\") pod \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\" (UID: \"f42a5beb-ad0f-43fc-97ad-0af6febd68ee\") " Oct 31 00:50:10.167921 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.167891 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-prometheus-data-science-monitoringstack-rulefiles-0" pod "f42a5beb-ad0f-43fc-97ad-0af6febd68ee" (UID: "f42a5beb-ad0f-43fc-97ad-0af6febd68ee"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:10.168043 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.167926 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-prometheus-data-science-monitoringstack-rulefiles-2" pod "f42a5beb-ad0f-43fc-97ad-0af6febd68ee" (UID: "f42a5beb-ad0f-43fc-97ad-0af6febd68ee"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:10.168233 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.168210 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-prometheus-data-science-monitoringstack-rulefiles-1" pod "f42a5beb-ad0f-43fc-97ad-0af6febd68ee" (UID: "f42a5beb-ad0f-43fc-97ad-0af6febd68ee"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:10.169228 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.169202 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-tls-assets" pod "f42a5beb-ad0f-43fc-97ad-0af6febd68ee" (UID: "f42a5beb-ad0f-43fc-97ad-0af6febd68ee"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:10.169640 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.169620 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-config-out" pod "f42a5beb-ad0f-43fc-97ad-0af6febd68ee" (UID: "f42a5beb-ad0f-43fc-97ad-0af6febd68ee"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:10.170183 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.170152 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "f42a5beb-ad0f-43fc-97ad-0af6febd68ee" (UID: "f42a5beb-ad0f-43fc-97ad-0af6febd68ee"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:10.170475 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.170448 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "f42a5beb-ad0f-43fc-97ad-0af6febd68ee" (UID: "f42a5beb-ad0f-43fc-97ad-0af6febd68ee"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:10.170588 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.170511 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-secret-prometheus-web-tls-ca" pod "f42a5beb-ad0f-43fc-97ad-0af6febd68ee" (UID: "f42a5beb-ad0f-43fc-97ad-0af6febd68ee"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:10.171457 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.171434 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-kube-api-access-gqvt4" pod "f42a5beb-ad0f-43fc-97ad-0af6febd68ee" (UID: "f42a5beb-ad0f-43fc-97ad-0af6febd68ee"). InnerVolumeSpecName "kube-api-access-gqvt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:10.171684 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.171666 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-thanos-prometheus-http-client-file" pod "f42a5beb-ad0f-43fc-97ad-0af6febd68ee" (UID: "f42a5beb-ad0f-43fc-97ad-0af6febd68ee"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:10.172260 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.172240 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a" pod "f42a5beb-ad0f-43fc-97ad-0af6febd68ee" (UID: "f42a5beb-ad0f-43fc-97ad-0af6febd68ee"). InnerVolumeSpecName "pvc-63e34709-8d85-4142-b53a-f655698606ff". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:50:10.178973 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.178897 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-config" pod "f42a5beb-ad0f-43fc-97ad-0af6febd68ee" (UID: "f42a5beb-ad0f-43fc-97ad-0af6febd68ee"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:10.183598 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.183571 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-web-config" pod "f42a5beb-ad0f-43fc-97ad-0af6febd68ee" (UID: "f42a5beb-ad0f-43fc-97ad-0af6febd68ee"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:10.243539 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.243356 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"976644b4-136a-42bd-95c8-c5dd1e415ac6","Type":"ContainerStarted","Data":"abedbafde1f9ad86f2d0734f0476815c6d2ef908da46c2fd2ecca3724614d4d4"} Oct 31 00:50:10.243539 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.243374 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="976644b4-136a-42bd-95c8-c5dd1e415ac6" containerName="otc-container" containerID="cri-o://abedbafde1f9ad86f2d0734f0476815c6d2ef908da46c2fd2ecca3724614d4d4" gracePeriod=30 Oct 31 00:50:10.243539 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.243422 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"976644b4-136a-42bd-95c8-c5dd1e415ac6","Type":"ContainerStarted","Data":"a2063e4cdf4b6b748cfdd16d1e13099ec8dcd01c5b5f326cf081a5e0670593a8"} Oct 31 00:50:10.247114 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.247087 5274 generic.go:356] "Generic (PLEG): container finished" podID="9777d542-77d4-4ad3-b2c5-a60b8ec24d68" containerID="83ff9e940a17ee7a30376240e8c2c4b7d90e39cbf718bdf619e1322ebde9ab47" exitCode=0 Oct 31 00:50:10.247264 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.247125 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"9777d542-77d4-4ad3-b2c5-a60b8ec24d68","Type":"ContainerDied","Data":"83ff9e940a17ee7a30376240e8c2c4b7d90e39cbf718bdf619e1322ebde9ab47"} Oct 31 00:50:10.248890 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.248573 5274 generic.go:356] "Generic (PLEG): container finished" podID="58d39fb2-6f91-4f59-aa7c-45e69e43cb12" containerID="fb76b61862f6ca9f45a389d10cfe5da76ea8b6356f3c4e58b5c93118b7e1298d" exitCode=0 Oct 31 00:50:10.248890 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.248616 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-499qg" Oct 31 00:50:10.248890 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.248652 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-499qg" event={"ID":"58d39fb2-6f91-4f59-aa7c-45e69e43cb12","Type":"ContainerDied","Data":"fb76b61862f6ca9f45a389d10cfe5da76ea8b6356f3c4e58b5c93118b7e1298d"} Oct 31 00:50:10.248890 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.248712 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-499qg" event={"ID":"58d39fb2-6f91-4f59-aa7c-45e69e43cb12","Type":"ContainerDied","Data":"6681a5641fc7b1d4f8726a539850356defe88b71a608522ae2841273e030e485"} Oct 31 00:50:10.248890 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.248739 5274 scope.go:122] "RemoveContainer" containerID="fb76b61862f6ca9f45a389d10cfe5da76ea8b6356f3c4e58b5c93118b7e1298d" Oct 31 00:50:10.250491 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.250465 5274 generic.go:356] "Generic (PLEG): container finished" podID="149823e9-b0e4-476c-9d37-d4581e7ea7d1" containerID="edb61927ba6edbbfebed49df327d5d7d7aaa15c5dc32ffedb9258645f93e5db8" exitCode=0 Oct 31 00:50:10.250616 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.250499 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-p2hvj" Oct 31 00:50:10.250616 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.250535 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-p2hvj" event={"ID":"149823e9-b0e4-476c-9d37-d4581e7ea7d1","Type":"ContainerDied","Data":"edb61927ba6edbbfebed49df327d5d7d7aaa15c5dc32ffedb9258645f93e5db8"} Oct 31 00:50:10.250616 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.250576 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-p2hvj" event={"ID":"149823e9-b0e4-476c-9d37-d4581e7ea7d1","Type":"ContainerDied","Data":"ea358c824d5748fcb9cb2cc4c3c1f2d6daf1dadcc49a634693b22a6bb3366ebf"} Oct 31 00:50:10.254180 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.254157 5274 generic.go:356] "Generic (PLEG): container finished" podID="f42a5beb-ad0f-43fc-97ad-0af6febd68ee" containerID="668c602e5308c4841913ecdfb7c4f1ba3c6b8fb5db6f39a0cb38f9087339d140" exitCode=0 Oct 31 00:50:10.254180 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.254179 5274 generic.go:356] "Generic (PLEG): container finished" podID="f42a5beb-ad0f-43fc-97ad-0af6febd68ee" containerID="373b52de232d2413f8ca9ed9494345c3a19b7f6f340b7f3e16aa5d0823d5c703" exitCode=0 Oct 31 00:50:10.254349 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.254188 5274 generic.go:356] "Generic (PLEG): container finished" podID="f42a5beb-ad0f-43fc-97ad-0af6febd68ee" containerID="7be120732babac11b637dcb826688d00ea943679e124ffd2358545eef3d76385" exitCode=0 Oct 31 00:50:10.254349 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.254211 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:10.254349 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.254209 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"f42a5beb-ad0f-43fc-97ad-0af6febd68ee","Type":"ContainerDied","Data":"668c602e5308c4841913ecdfb7c4f1ba3c6b8fb5db6f39a0cb38f9087339d140"} Oct 31 00:50:10.254505 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.254348 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"f42a5beb-ad0f-43fc-97ad-0af6febd68ee","Type":"ContainerDied","Data":"373b52de232d2413f8ca9ed9494345c3a19b7f6f340b7f3e16aa5d0823d5c703"} Oct 31 00:50:10.254505 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.254393 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"f42a5beb-ad0f-43fc-97ad-0af6febd68ee","Type":"ContainerDied","Data":"7be120732babac11b637dcb826688d00ea943679e124ffd2358545eef3d76385"} Oct 31 00:50:10.254505 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.254430 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"f42a5beb-ad0f-43fc-97ad-0af6febd68ee","Type":"ContainerDied","Data":"8d3deeac9c8c68e4df134436e2c9e5a7d156a1a1351124d9f978139f3b6593fc"} Oct 31 00:50:10.265290 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.265256 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.265242152 podStartE2EDuration="1.265242152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:10.263940985 +0000 UTC m=+3046.900759699" watchObservedRunningTime="2026-10-31 00:50:10.265242152 +0000 UTC m=+3046.902060870" Oct 31 00:50:10.267460 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.267441 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.267556 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.267462 5274 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.267556 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.267475 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqvt4\" (UniqueName: \"kubernetes.io/projected/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-kube-api-access-gqvt4\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.267556 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.267484 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.267556 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.267495 5274 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.267556 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.267509 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.267556 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.267519 5274 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-secret-prometheus-web-tls-ca\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.267556 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.267529 5274 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.267556 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.267556 5274 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" " Oct 31 00:50:10.268055 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.267567 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.268055 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.267576 5274 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-tls-assets\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.268055 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.267589 5274 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-config-out\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.268055 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.267601 5274 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f42a5beb-ad0f-43fc-97ad-0af6febd68ee-web-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.278576 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.278560 5274 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-63e34709-8d85-4142-b53a-f655698606ff" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a") on node "ip-10-0-129-70.us-east-2.compute.internal" Oct 31 00:50:10.300452 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.300437 5274 scope.go:122] "RemoveContainer" containerID="fb76b61862f6ca9f45a389d10cfe5da76ea8b6356f3c4e58b5c93118b7e1298d" Oct 31 00:50:10.300889 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:50:10.300797 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb76b61862f6ca9f45a389d10cfe5da76ea8b6356f3c4e58b5c93118b7e1298d\": container with ID starting with fb76b61862f6ca9f45a389d10cfe5da76ea8b6356f3c4e58b5c93118b7e1298d not found: ID does not exist" containerID="fb76b61862f6ca9f45a389d10cfe5da76ea8b6356f3c4e58b5c93118b7e1298d" Oct 31 00:50:10.300889 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.300830 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb76b61862f6ca9f45a389d10cfe5da76ea8b6356f3c4e58b5c93118b7e1298d"} err="failed to get container status \"fb76b61862f6ca9f45a389d10cfe5da76ea8b6356f3c4e58b5c93118b7e1298d\": rpc error: code = NotFound desc = could not find container \"fb76b61862f6ca9f45a389d10cfe5da76ea8b6356f3c4e58b5c93118b7e1298d\": container with ID starting with fb76b61862f6ca9f45a389d10cfe5da76ea8b6356f3c4e58b5c93118b7e1298d not found: ID does not exist" Oct 31 00:50:10.300889 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.300850 5274 scope.go:122] "RemoveContainer" containerID="edb61927ba6edbbfebed49df327d5d7d7aaa15c5dc32ffedb9258645f93e5db8" Oct 31 00:50:10.318487 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.318461 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-499qg"] Oct 31 00:50:10.320432 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.320410 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-499qg"] Oct 31 00:50:10.332569 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.332545 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-p2hvj"] Oct 31 00:50:10.337425 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.337406 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-p2hvj"] Oct 31 00:50:10.354443 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.354422 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:50:10.355924 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.355907 5274 scope.go:122] "RemoveContainer" containerID="edb61927ba6edbbfebed49df327d5d7d7aaa15c5dc32ffedb9258645f93e5db8" Oct 31 00:50:10.356481 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:50:10.356380 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edb61927ba6edbbfebed49df327d5d7d7aaa15c5dc32ffedb9258645f93e5db8\": container with ID starting with edb61927ba6edbbfebed49df327d5d7d7aaa15c5dc32ffedb9258645f93e5db8 not found: ID does not exist" containerID="edb61927ba6edbbfebed49df327d5d7d7aaa15c5dc32ffedb9258645f93e5db8" Oct 31 00:50:10.356481 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.356403 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edb61927ba6edbbfebed49df327d5d7d7aaa15c5dc32ffedb9258645f93e5db8"} err="failed to get container status \"edb61927ba6edbbfebed49df327d5d7d7aaa15c5dc32ffedb9258645f93e5db8\": rpc error: code = NotFound desc = could not find container \"edb61927ba6edbbfebed49df327d5d7d7aaa15c5dc32ffedb9258645f93e5db8\": container with ID starting with edb61927ba6edbbfebed49df327d5d7d7aaa15c5dc32ffedb9258645f93e5db8 not found: ID does not exist" Oct 31 00:50:10.356481 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.356419 5274 scope.go:122] "RemoveContainer" containerID="668c602e5308c4841913ecdfb7c4f1ba3c6b8fb5db6f39a0cb38f9087339d140" Oct 31 00:50:10.359603 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.359587 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:50:10.368508 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.368487 5274 reconciler_common.go:299] "Volume detached for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"csi-3beb91b3ebf177571ca474589da3560cd18787cfa50b2a0b9bfef85661e6cf21\"" Oct 31 00:50:10.369932 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.369907 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:50:10.372257 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.372223 5274 scope.go:122] "RemoveContainer" containerID="373b52de232d2413f8ca9ed9494345c3a19b7f6f340b7f3e16aa5d0823d5c703" Oct 31 00:50:10.380694 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.380672 5274 scope.go:122] "RemoveContainer" containerID="7be120732babac11b637dcb826688d00ea943679e124ffd2358545eef3d76385" Oct 31 00:50:10.388679 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.388663 5274 scope.go:122] "RemoveContainer" containerID="b749b49dad5719aff95215bfcc3425c768d9580ba426f0558c636d3ca8cb67b4" Oct 31 00:50:10.399326 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.399309 5274 scope.go:122] "RemoveContainer" containerID="668c602e5308c4841913ecdfb7c4f1ba3c6b8fb5db6f39a0cb38f9087339d140" Oct 31 00:50:10.399629 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:50:10.399603 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"668c602e5308c4841913ecdfb7c4f1ba3c6b8fb5db6f39a0cb38f9087339d140\": container with ID starting with 668c602e5308c4841913ecdfb7c4f1ba3c6b8fb5db6f39a0cb38f9087339d140 not found: ID does not exist" containerID="668c602e5308c4841913ecdfb7c4f1ba3c6b8fb5db6f39a0cb38f9087339d140" Oct 31 00:50:10.399724 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.399637 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"668c602e5308c4841913ecdfb7c4f1ba3c6b8fb5db6f39a0cb38f9087339d140"} err="failed to get container status \"668c602e5308c4841913ecdfb7c4f1ba3c6b8fb5db6f39a0cb38f9087339d140\": rpc error: code = NotFound desc = could not find container \"668c602e5308c4841913ecdfb7c4f1ba3c6b8fb5db6f39a0cb38f9087339d140\": container with ID starting with 668c602e5308c4841913ecdfb7c4f1ba3c6b8fb5db6f39a0cb38f9087339d140 not found: ID does not exist" Oct 31 00:50:10.399724 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.399656 5274 scope.go:122] "RemoveContainer" containerID="373b52de232d2413f8ca9ed9494345c3a19b7f6f340b7f3e16aa5d0823d5c703" Oct 31 00:50:10.400058 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:50:10.400033 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"373b52de232d2413f8ca9ed9494345c3a19b7f6f340b7f3e16aa5d0823d5c703\": container with ID starting with 373b52de232d2413f8ca9ed9494345c3a19b7f6f340b7f3e16aa5d0823d5c703 not found: ID does not exist" containerID="373b52de232d2413f8ca9ed9494345c3a19b7f6f340b7f3e16aa5d0823d5c703" Oct 31 00:50:10.400143 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.400063 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"373b52de232d2413f8ca9ed9494345c3a19b7f6f340b7f3e16aa5d0823d5c703"} err="failed to get container status \"373b52de232d2413f8ca9ed9494345c3a19b7f6f340b7f3e16aa5d0823d5c703\": rpc error: code = NotFound desc = could not find container \"373b52de232d2413f8ca9ed9494345c3a19b7f6f340b7f3e16aa5d0823d5c703\": container with ID starting with 373b52de232d2413f8ca9ed9494345c3a19b7f6f340b7f3e16aa5d0823d5c703 not found: ID does not exist" Oct 31 00:50:10.400143 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.400079 5274 scope.go:122] "RemoveContainer" containerID="7be120732babac11b637dcb826688d00ea943679e124ffd2358545eef3d76385" Oct 31 00:50:10.400393 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:50:10.400365 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7be120732babac11b637dcb826688d00ea943679e124ffd2358545eef3d76385\": container with ID starting with 7be120732babac11b637dcb826688d00ea943679e124ffd2358545eef3d76385 not found: ID does not exist" containerID="7be120732babac11b637dcb826688d00ea943679e124ffd2358545eef3d76385" Oct 31 00:50:10.400468 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.400393 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7be120732babac11b637dcb826688d00ea943679e124ffd2358545eef3d76385"} err="failed to get container status \"7be120732babac11b637dcb826688d00ea943679e124ffd2358545eef3d76385\": rpc error: code = NotFound desc = could not find container \"7be120732babac11b637dcb826688d00ea943679e124ffd2358545eef3d76385\": container with ID starting with 7be120732babac11b637dcb826688d00ea943679e124ffd2358545eef3d76385 not found: ID does not exist" Oct 31 00:50:10.400468 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.400415 5274 scope.go:122] "RemoveContainer" containerID="b749b49dad5719aff95215bfcc3425c768d9580ba426f0558c636d3ca8cb67b4" Oct 31 00:50:10.400697 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:50:10.400675 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b749b49dad5719aff95215bfcc3425c768d9580ba426f0558c636d3ca8cb67b4\": container with ID starting with b749b49dad5719aff95215bfcc3425c768d9580ba426f0558c636d3ca8cb67b4 not found: ID does not exist" containerID="b749b49dad5719aff95215bfcc3425c768d9580ba426f0558c636d3ca8cb67b4" Oct 31 00:50:10.400742 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.400695 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b749b49dad5719aff95215bfcc3425c768d9580ba426f0558c636d3ca8cb67b4"} err="failed to get container status \"b749b49dad5719aff95215bfcc3425c768d9580ba426f0558c636d3ca8cb67b4\": rpc error: code = NotFound desc = could not find container \"b749b49dad5719aff95215bfcc3425c768d9580ba426f0558c636d3ca8cb67b4\": container with ID starting with b749b49dad5719aff95215bfcc3425c768d9580ba426f0558c636d3ca8cb67b4 not found: ID does not exist" Oct 31 00:50:10.400742 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.400714 5274 scope.go:122] "RemoveContainer" containerID="668c602e5308c4841913ecdfb7c4f1ba3c6b8fb5db6f39a0cb38f9087339d140" Oct 31 00:50:10.400981 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.400935 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"668c602e5308c4841913ecdfb7c4f1ba3c6b8fb5db6f39a0cb38f9087339d140"} err="failed to get container status \"668c602e5308c4841913ecdfb7c4f1ba3c6b8fb5db6f39a0cb38f9087339d140\": rpc error: code = NotFound desc = could not find container \"668c602e5308c4841913ecdfb7c4f1ba3c6b8fb5db6f39a0cb38f9087339d140\": container with ID starting with 668c602e5308c4841913ecdfb7c4f1ba3c6b8fb5db6f39a0cb38f9087339d140 not found: ID does not exist" Oct 31 00:50:10.401056 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.400992 5274 scope.go:122] "RemoveContainer" containerID="373b52de232d2413f8ca9ed9494345c3a19b7f6f340b7f3e16aa5d0823d5c703" Oct 31 00:50:10.401287 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.401248 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"373b52de232d2413f8ca9ed9494345c3a19b7f6f340b7f3e16aa5d0823d5c703"} err="failed to get container status \"373b52de232d2413f8ca9ed9494345c3a19b7f6f340b7f3e16aa5d0823d5c703\": rpc error: code = NotFound desc = could not find container \"373b52de232d2413f8ca9ed9494345c3a19b7f6f340b7f3e16aa5d0823d5c703\": container with ID starting with 373b52de232d2413f8ca9ed9494345c3a19b7f6f340b7f3e16aa5d0823d5c703 not found: ID does not exist" Oct 31 00:50:10.401287 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.401276 5274 scope.go:122] "RemoveContainer" containerID="7be120732babac11b637dcb826688d00ea943679e124ffd2358545eef3d76385" Oct 31 00:50:10.401525 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.401508 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7be120732babac11b637dcb826688d00ea943679e124ffd2358545eef3d76385"} err="failed to get container status \"7be120732babac11b637dcb826688d00ea943679e124ffd2358545eef3d76385\": rpc error: code = NotFound desc = could not find container \"7be120732babac11b637dcb826688d00ea943679e124ffd2358545eef3d76385\": container with ID starting with 7be120732babac11b637dcb826688d00ea943679e124ffd2358545eef3d76385 not found: ID does not exist" Oct 31 00:50:10.401525 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.401524 5274 scope.go:122] "RemoveContainer" containerID="b749b49dad5719aff95215bfcc3425c768d9580ba426f0558c636d3ca8cb67b4" Oct 31 00:50:10.401803 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.401782 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b749b49dad5719aff95215bfcc3425c768d9580ba426f0558c636d3ca8cb67b4"} err="failed to get container status \"b749b49dad5719aff95215bfcc3425c768d9580ba426f0558c636d3ca8cb67b4\": rpc error: code = NotFound desc = could not find container \"b749b49dad5719aff95215bfcc3425c768d9580ba426f0558c636d3ca8cb67b4\": container with ID starting with b749b49dad5719aff95215bfcc3425c768d9580ba426f0558c636d3ca8cb67b4 not found: ID does not exist" Oct 31 00:50:10.401803 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.401801 5274 scope.go:122] "RemoveContainer" containerID="668c602e5308c4841913ecdfb7c4f1ba3c6b8fb5db6f39a0cb38f9087339d140" Oct 31 00:50:10.402253 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.402090 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"668c602e5308c4841913ecdfb7c4f1ba3c6b8fb5db6f39a0cb38f9087339d140"} err="failed to get container status \"668c602e5308c4841913ecdfb7c4f1ba3c6b8fb5db6f39a0cb38f9087339d140\": rpc error: code = NotFound desc = could not find container \"668c602e5308c4841913ecdfb7c4f1ba3c6b8fb5db6f39a0cb38f9087339d140\": container with ID starting with 668c602e5308c4841913ecdfb7c4f1ba3c6b8fb5db6f39a0cb38f9087339d140 not found: ID does not exist" Oct 31 00:50:10.402253 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.402116 5274 scope.go:122] "RemoveContainer" containerID="373b52de232d2413f8ca9ed9494345c3a19b7f6f340b7f3e16aa5d0823d5c703" Oct 31 00:50:10.402654 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.402473 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"373b52de232d2413f8ca9ed9494345c3a19b7f6f340b7f3e16aa5d0823d5c703"} err="failed to get container status \"373b52de232d2413f8ca9ed9494345c3a19b7f6f340b7f3e16aa5d0823d5c703\": rpc error: code = NotFound desc = could not find container \"373b52de232d2413f8ca9ed9494345c3a19b7f6f340b7f3e16aa5d0823d5c703\": container with ID starting with 373b52de232d2413f8ca9ed9494345c3a19b7f6f340b7f3e16aa5d0823d5c703 not found: ID does not exist" Oct 31 00:50:10.402654 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.402496 5274 scope.go:122] "RemoveContainer" containerID="7be120732babac11b637dcb826688d00ea943679e124ffd2358545eef3d76385" Oct 31 00:50:10.402815 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.402792 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7be120732babac11b637dcb826688d00ea943679e124ffd2358545eef3d76385"} err="failed to get container status \"7be120732babac11b637dcb826688d00ea943679e124ffd2358545eef3d76385\": rpc error: code = NotFound desc = could not find container \"7be120732babac11b637dcb826688d00ea943679e124ffd2358545eef3d76385\": container with ID starting with 7be120732babac11b637dcb826688d00ea943679e124ffd2358545eef3d76385 not found: ID does not exist" Oct 31 00:50:10.402865 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.402817 5274 scope.go:122] "RemoveContainer" containerID="b749b49dad5719aff95215bfcc3425c768d9580ba426f0558c636d3ca8cb67b4" Oct 31 00:50:10.403148 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.403108 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b749b49dad5719aff95215bfcc3425c768d9580ba426f0558c636d3ca8cb67b4"} err="failed to get container status \"b749b49dad5719aff95215bfcc3425c768d9580ba426f0558c636d3ca8cb67b4\": rpc error: code = NotFound desc = could not find container \"b749b49dad5719aff95215bfcc3425c768d9580ba426f0558c636d3ca8cb67b4\": container with ID starting with b749b49dad5719aff95215bfcc3425c768d9580ba426f0558c636d3ca8cb67b4 not found: ID does not exist" Oct 31 00:50:10.469637 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.469583 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-web-config\" (UniqueName: \"kubernetes.io/secret/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-web-config\") pod \"9777d542-77d4-4ad3-b2c5-a60b8ec24d68\" (UID: \"9777d542-77d4-4ad3-b2c5-a60b8ec24d68\") " Oct 31 00:50:10.469637 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.469620 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-config-out\" (UniqueName: \"kubernetes.io/empty-dir/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-config-out\") pod \"9777d542-77d4-4ad3-b2c5-a60b8ec24d68\" (UID: \"9777d542-77d4-4ad3-b2c5-a60b8ec24d68\") " Oct 31 00:50:10.469813 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.469658 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-kube-api-access-4kgj5\" (UniqueName: \"kubernetes.io/projected/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-kube-api-access-4kgj5\") pod \"9777d542-77d4-4ad3-b2c5-a60b8ec24d68\" (UID: \"9777d542-77d4-4ad3-b2c5-a60b8ec24d68\") " Oct 31 00:50:10.469813 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.469730 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-cluster-tls-config\") pod \"9777d542-77d4-4ad3-b2c5-a60b8ec24d68\" (UID: \"9777d542-77d4-4ad3-b2c5-a60b8ec24d68\") " Oct 31 00:50:10.469813 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.469791 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-config-volume\" (UniqueName: \"kubernetes.io/secret/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-config-volume\") pod \"9777d542-77d4-4ad3-b2c5-a60b8ec24d68\" (UID: \"9777d542-77d4-4ad3-b2c5-a60b8ec24d68\") " Oct 31 00:50:10.470040 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.469851 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-alertmanager-data-science-monitoringstack-db\") pod \"9777d542-77d4-4ad3-b2c5-a60b8ec24d68\" (UID: \"9777d542-77d4-4ad3-b2c5-a60b8ec24d68\") " Oct 31 00:50:10.470040 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.469906 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-tls-assets\" (UniqueName: \"kubernetes.io/projected/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-tls-assets\") pod \"9777d542-77d4-4ad3-b2c5-a60b8ec24d68\" (UID: \"9777d542-77d4-4ad3-b2c5-a60b8ec24d68\") " Oct 31 00:50:10.470231 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.470203 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-alertmanager-data-science-monitoringstack-db" pod "9777d542-77d4-4ad3-b2c5-a60b8ec24d68" (UID: "9777d542-77d4-4ad3-b2c5-a60b8ec24d68"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:10.472600 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.472407 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-config-volume" pod "9777d542-77d4-4ad3-b2c5-a60b8ec24d68" (UID: "9777d542-77d4-4ad3-b2c5-a60b8ec24d68"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:10.472600 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.472498 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-tls-assets" pod "9777d542-77d4-4ad3-b2c5-a60b8ec24d68" (UID: "9777d542-77d4-4ad3-b2c5-a60b8ec24d68"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:10.472600 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.472533 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-config-out" pod "9777d542-77d4-4ad3-b2c5-a60b8ec24d68" (UID: "9777d542-77d4-4ad3-b2c5-a60b8ec24d68"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:10.472600 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.472579 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-kube-api-access-4kgj5" pod "9777d542-77d4-4ad3-b2c5-a60b8ec24d68" (UID: "9777d542-77d4-4ad3-b2c5-a60b8ec24d68"). InnerVolumeSpecName "kube-api-access-4kgj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:10.477475 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.477450 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-cluster-tls-config" pod "9777d542-77d4-4ad3-b2c5-a60b8ec24d68" (UID: "9777d542-77d4-4ad3-b2c5-a60b8ec24d68"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:10.482867 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.482847 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-web-config" pod "9777d542-77d4-4ad3-b2c5-a60b8ec24d68" (UID: "9777d542-77d4-4ad3-b2c5-a60b8ec24d68"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:10.492085 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.492067 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:10.570636 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.570598 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/976644b4-136a-42bd-95c8-c5dd1e415ac6-otc-internal\" (UniqueName: \"kubernetes.io/configmap/976644b4-136a-42bd-95c8-c5dd1e415ac6-otc-internal\") pod \"976644b4-136a-42bd-95c8-c5dd1e415ac6\" (UID: \"976644b4-136a-42bd-95c8-c5dd1e415ac6\") " Oct 31 00:50:10.570809 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.570670 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/976644b4-136a-42bd-95c8-c5dd1e415ac6-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/976644b4-136a-42bd-95c8-c5dd1e415ac6-otlp-tls-certs\") pod \"976644b4-136a-42bd-95c8-c5dd1e415ac6\" (UID: \"976644b4-136a-42bd-95c8-c5dd1e415ac6\") " Oct 31 00:50:10.570809 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.570717 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/976644b4-136a-42bd-95c8-c5dd1e415ac6-tls-certs\" (UniqueName: \"kubernetes.io/secret/976644b4-136a-42bd-95c8-c5dd1e415ac6-tls-certs\") pod \"976644b4-136a-42bd-95c8-c5dd1e415ac6\" (UID: \"976644b4-136a-42bd-95c8-c5dd1e415ac6\") " Oct 31 00:50:10.570809 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.570764 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/976644b4-136a-42bd-95c8-c5dd1e415ac6-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/976644b4-136a-42bd-95c8-c5dd1e415ac6-data-science-collector-ta-client-cert\") pod \"976644b4-136a-42bd-95c8-c5dd1e415ac6\" (UID: \"976644b4-136a-42bd-95c8-c5dd1e415ac6\") " Oct 31 00:50:10.570809 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.570790 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/976644b4-136a-42bd-95c8-c5dd1e415ac6-kube-api-access-8vpkz\" (UniqueName: \"kubernetes.io/projected/976644b4-136a-42bd-95c8-c5dd1e415ac6-kube-api-access-8vpkz\") pod \"976644b4-136a-42bd-95c8-c5dd1e415ac6\" (UID: \"976644b4-136a-42bd-95c8-c5dd1e415ac6\") " Oct 31 00:50:10.571064 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.570815 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/976644b4-136a-42bd-95c8-c5dd1e415ac6-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/976644b4-136a-42bd-95c8-c5dd1e415ac6-monitoring-tls-certs\") pod \"976644b4-136a-42bd-95c8-c5dd1e415ac6\" (UID: \"976644b4-136a-42bd-95c8-c5dd1e415ac6\") " Oct 31 00:50:10.571064 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.571053 5274 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-config-volume\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.571176 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.571076 5274 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.571176 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.571082 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/976644b4-136a-42bd-95c8-c5dd1e415ac6-otc-internal" pod "976644b4-136a-42bd-95c8-c5dd1e415ac6" (UID: "976644b4-136a-42bd-95c8-c5dd1e415ac6"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:10.571176 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.571093 5274 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-tls-assets\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.571176 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.571146 5274 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-web-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.571176 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.571167 5274 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-config-out\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.571449 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.571184 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kgj5\" (UniqueName: \"kubernetes.io/projected/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-kube-api-access-4kgj5\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.571449 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.571205 5274 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9777d542-77d4-4ad3-b2c5-a60b8ec24d68-cluster-tls-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.573023 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.572992 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/976644b4-136a-42bd-95c8-c5dd1e415ac6-otlp-tls-certs" pod "976644b4-136a-42bd-95c8-c5dd1e415ac6" (UID: "976644b4-136a-42bd-95c8-c5dd1e415ac6"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:10.573151 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.573044 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/976644b4-136a-42bd-95c8-c5dd1e415ac6-kube-api-access-8vpkz" pod "976644b4-136a-42bd-95c8-c5dd1e415ac6" (UID: "976644b4-136a-42bd-95c8-c5dd1e415ac6"). InnerVolumeSpecName "kube-api-access-8vpkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:10.573151 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.573071 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/976644b4-136a-42bd-95c8-c5dd1e415ac6-tls-certs" pod "976644b4-136a-42bd-95c8-c5dd1e415ac6" (UID: "976644b4-136a-42bd-95c8-c5dd1e415ac6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:10.573356 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.573339 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/976644b4-136a-42bd-95c8-c5dd1e415ac6-monitoring-tls-certs" pod "976644b4-136a-42bd-95c8-c5dd1e415ac6" (UID: "976644b4-136a-42bd-95c8-c5dd1e415ac6"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:10.573404 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.573372 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/976644b4-136a-42bd-95c8-c5dd1e415ac6-data-science-collector-ta-client-cert" pod "976644b4-136a-42bd-95c8-c5dd1e415ac6" (UID: "976644b4-136a-42bd-95c8-c5dd1e415ac6"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:10.671922 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.671876 5274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/976644b4-136a-42bd-95c8-c5dd1e415ac6-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.671922 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.671918 5274 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/976644b4-136a-42bd-95c8-c5dd1e415ac6-data-science-collector-ta-client-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.671922 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.671932 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vpkz\" (UniqueName: \"kubernetes.io/projected/976644b4-136a-42bd-95c8-c5dd1e415ac6-kube-api-access-8vpkz\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.672189 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.671946 5274 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/976644b4-136a-42bd-95c8-c5dd1e415ac6-monitoring-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.672189 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.671985 5274 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/976644b4-136a-42bd-95c8-c5dd1e415ac6-otc-internal\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.672189 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:10.671998 5274 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/976644b4-136a-42bd-95c8-c5dd1e415ac6-otlp-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:11.261356 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:11.261269 5274 generic.go:356] "Generic (PLEG): container finished" podID="976644b4-136a-42bd-95c8-c5dd1e415ac6" containerID="abedbafde1f9ad86f2d0734f0476815c6d2ef908da46c2fd2ecca3724614d4d4" exitCode=2 Oct 31 00:50:11.261356 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:11.261318 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:11.261853 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:11.261358 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"976644b4-136a-42bd-95c8-c5dd1e415ac6","Type":"ContainerDied","Data":"abedbafde1f9ad86f2d0734f0476815c6d2ef908da46c2fd2ecca3724614d4d4"} Oct 31 00:50:11.261853 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:11.261411 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"976644b4-136a-42bd-95c8-c5dd1e415ac6","Type":"ContainerDied","Data":"a2063e4cdf4b6b748cfdd16d1e13099ec8dcd01c5b5f326cf081a5e0670593a8"} Oct 31 00:50:11.261853 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:11.261440 5274 scope.go:122] "RemoveContainer" containerID="abedbafde1f9ad86f2d0734f0476815c6d2ef908da46c2fd2ecca3724614d4d4" Oct 31 00:50:11.263621 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:11.263596 5274 generic.go:356] "Generic (PLEG): container finished" podID="9777d542-77d4-4ad3-b2c5-a60b8ec24d68" containerID="76180109ab66e857297d5108073731e23f4f2fe16cce995d736a8bd191c0c61b" exitCode=0 Oct 31 00:50:11.263726 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:11.263651 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:50:11.263726 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:11.263699 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"9777d542-77d4-4ad3-b2c5-a60b8ec24d68","Type":"ContainerDied","Data":"76180109ab66e857297d5108073731e23f4f2fe16cce995d736a8bd191c0c61b"} Oct 31 00:50:11.263832 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:11.263744 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"9777d542-77d4-4ad3-b2c5-a60b8ec24d68","Type":"ContainerDied","Data":"346f59ca0b528b5d407d011f254d8e9adc952557d0beeccb14e7be712072371a"} Oct 31 00:50:11.271404 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:11.271386 5274 scope.go:122] "RemoveContainer" containerID="abedbafde1f9ad86f2d0734f0476815c6d2ef908da46c2fd2ecca3724614d4d4" Oct 31 00:50:11.271929 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:50:11.271667 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abedbafde1f9ad86f2d0734f0476815c6d2ef908da46c2fd2ecca3724614d4d4\": container with ID starting with abedbafde1f9ad86f2d0734f0476815c6d2ef908da46c2fd2ecca3724614d4d4 not found: ID does not exist" containerID="abedbafde1f9ad86f2d0734f0476815c6d2ef908da46c2fd2ecca3724614d4d4" Oct 31 00:50:11.271929 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:11.271693 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abedbafde1f9ad86f2d0734f0476815c6d2ef908da46c2fd2ecca3724614d4d4"} err="failed to get container status \"abedbafde1f9ad86f2d0734f0476815c6d2ef908da46c2fd2ecca3724614d4d4\": rpc error: code = NotFound desc = could not find container \"abedbafde1f9ad86f2d0734f0476815c6d2ef908da46c2fd2ecca3724614d4d4\": container with ID starting with abedbafde1f9ad86f2d0734f0476815c6d2ef908da46c2fd2ecca3724614d4d4 not found: ID does not exist" Oct 31 00:50:11.271929 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:11.271710 5274 scope.go:122] "RemoveContainer" containerID="83ff9e940a17ee7a30376240e8c2c4b7d90e39cbf718bdf619e1322ebde9ab47" Oct 31 00:50:11.279587 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:11.279572 5274 scope.go:122] "RemoveContainer" containerID="76180109ab66e857297d5108073731e23f4f2fe16cce995d736a8bd191c0c61b" Oct 31 00:50:11.287447 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:11.287430 5274 scope.go:122] "RemoveContainer" containerID="34e4ff05d91764d794fc49cfa76ffa6ff6184ae3b9b126a4a724f2e0e49acc32" Oct 31 00:50:11.289047 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:11.289030 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:50:11.293544 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:11.293518 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:50:11.295541 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:11.295526 5274 scope.go:122] "RemoveContainer" containerID="83ff9e940a17ee7a30376240e8c2c4b7d90e39cbf718bdf619e1322ebde9ab47" Oct 31 00:50:11.295778 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:50:11.295762 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83ff9e940a17ee7a30376240e8c2c4b7d90e39cbf718bdf619e1322ebde9ab47\": container with ID starting with 83ff9e940a17ee7a30376240e8c2c4b7d90e39cbf718bdf619e1322ebde9ab47 not found: ID does not exist" containerID="83ff9e940a17ee7a30376240e8c2c4b7d90e39cbf718bdf619e1322ebde9ab47" Oct 31 00:50:11.295823 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:11.295784 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ff9e940a17ee7a30376240e8c2c4b7d90e39cbf718bdf619e1322ebde9ab47"} err="failed to get container status \"83ff9e940a17ee7a30376240e8c2c4b7d90e39cbf718bdf619e1322ebde9ab47\": rpc error: code = NotFound desc = could not find container \"83ff9e940a17ee7a30376240e8c2c4b7d90e39cbf718bdf619e1322ebde9ab47\": container with ID starting with 83ff9e940a17ee7a30376240e8c2c4b7d90e39cbf718bdf619e1322ebde9ab47 not found: ID does not exist" Oct 31 00:50:11.295823 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:11.295798 5274 scope.go:122] "RemoveContainer" containerID="76180109ab66e857297d5108073731e23f4f2fe16cce995d736a8bd191c0c61b" Oct 31 00:50:11.296080 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:50:11.296060 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76180109ab66e857297d5108073731e23f4f2fe16cce995d736a8bd191c0c61b\": container with ID starting with 76180109ab66e857297d5108073731e23f4f2fe16cce995d736a8bd191c0c61b not found: ID does not exist" containerID="76180109ab66e857297d5108073731e23f4f2fe16cce995d736a8bd191c0c61b" Oct 31 00:50:11.296140 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:11.296087 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76180109ab66e857297d5108073731e23f4f2fe16cce995d736a8bd191c0c61b"} err="failed to get container status \"76180109ab66e857297d5108073731e23f4f2fe16cce995d736a8bd191c0c61b\": rpc error: code = NotFound desc = could not find container \"76180109ab66e857297d5108073731e23f4f2fe16cce995d736a8bd191c0c61b\": container with ID starting with 76180109ab66e857297d5108073731e23f4f2fe16cce995d736a8bd191c0c61b not found: ID does not exist" Oct 31 00:50:11.296140 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:11.296102 5274 scope.go:122] "RemoveContainer" containerID="34e4ff05d91764d794fc49cfa76ffa6ff6184ae3b9b126a4a724f2e0e49acc32" Oct 31 00:50:11.296350 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:50:11.296325 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34e4ff05d91764d794fc49cfa76ffa6ff6184ae3b9b126a4a724f2e0e49acc32\": container with ID starting with 34e4ff05d91764d794fc49cfa76ffa6ff6184ae3b9b126a4a724f2e0e49acc32 not found: ID does not exist" containerID="34e4ff05d91764d794fc49cfa76ffa6ff6184ae3b9b126a4a724f2e0e49acc32" Oct 31 00:50:11.296396 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:11.296345 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34e4ff05d91764d794fc49cfa76ffa6ff6184ae3b9b126a4a724f2e0e49acc32"} err="failed to get container status \"34e4ff05d91764d794fc49cfa76ffa6ff6184ae3b9b126a4a724f2e0e49acc32\": rpc error: code = NotFound desc = could not find container \"34e4ff05d91764d794fc49cfa76ffa6ff6184ae3b9b126a4a724f2e0e49acc32\": container with ID starting with 34e4ff05d91764d794fc49cfa76ffa6ff6184ae3b9b126a4a724f2e0e49acc32 not found: ID does not exist" Oct 31 00:50:11.307097 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:11.307069 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:50:11.312097 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:11.312070 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:50:11.480429 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:11.480396 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="149823e9-b0e4-476c-9d37-d4581e7ea7d1" path="/var/lib/kubelet/pods/149823e9-b0e4-476c-9d37-d4581e7ea7d1/volumes" Oct 31 00:50:11.480763 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:11.480750 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58d39fb2-6f91-4f59-aa7c-45e69e43cb12" path="/var/lib/kubelet/pods/58d39fb2-6f91-4f59-aa7c-45e69e43cb12/volumes" Oct 31 00:50:11.481104 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:11.481092 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="976644b4-136a-42bd-95c8-c5dd1e415ac6" path="/var/lib/kubelet/pods/976644b4-136a-42bd-95c8-c5dd1e415ac6/volumes" Oct 31 00:50:11.481698 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:11.481685 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9777d542-77d4-4ad3-b2c5-a60b8ec24d68" path="/var/lib/kubelet/pods/9777d542-77d4-4ad3-b2c5-a60b8ec24d68/volumes" Oct 31 00:50:11.482208 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:11.482195 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f42a5beb-ad0f-43fc-97ad-0af6febd68ee" path="/var/lib/kubelet/pods/f42a5beb-ad0f-43fc-97ad-0af6febd68ee/volumes" Oct 31 00:50:20.033218 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.033184 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-fmzfj"] Oct 31 00:50:20.033591 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.033516 5274 cpu_manager.go:414] "containerMap: removing container" podUID="f42a5beb-ad0f-43fc-97ad-0af6febd68ee" containerName="config-reloader" Oct 31 00:50:20.033591 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.033526 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f42a5beb-ad0f-43fc-97ad-0af6febd68ee" containerName="config-reloader" Oct 31 00:50:20.033591 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.033537 5274 cpu_manager.go:414] "containerMap: removing container" podUID="f42a5beb-ad0f-43fc-97ad-0af6febd68ee" containerName="init-config-reloader" Oct 31 00:50:20.033591 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.033542 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f42a5beb-ad0f-43fc-97ad-0af6febd68ee" containerName="init-config-reloader" Oct 31 00:50:20.033591 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.033554 5274 cpu_manager.go:414] "containerMap: removing container" podUID="f42a5beb-ad0f-43fc-97ad-0af6febd68ee" containerName="prometheus" Oct 31 00:50:20.033591 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.033559 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f42a5beb-ad0f-43fc-97ad-0af6febd68ee" containerName="prometheus" Oct 31 00:50:20.033591 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.033570 5274 cpu_manager.go:414] "containerMap: removing container" podUID="58d39fb2-6f91-4f59-aa7c-45e69e43cb12" containerName="korrel8r" Oct 31 00:50:20.033591 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.033575 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58d39fb2-6f91-4f59-aa7c-45e69e43cb12" containerName="korrel8r" Oct 31 00:50:20.033591 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.033580 5274 cpu_manager.go:414] "containerMap: removing container" podUID="9777d542-77d4-4ad3-b2c5-a60b8ec24d68" containerName="alertmanager" Oct 31 00:50:20.033591 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.033586 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9777d542-77d4-4ad3-b2c5-a60b8ec24d68" containerName="alertmanager" Oct 31 00:50:20.033591 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.033592 5274 cpu_manager.go:414] "containerMap: removing container" podUID="9777d542-77d4-4ad3-b2c5-a60b8ec24d68" containerName="init-config-reloader" Oct 31 00:50:20.033591 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.033597 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9777d542-77d4-4ad3-b2c5-a60b8ec24d68" containerName="init-config-reloader" Oct 31 00:50:20.034038 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.033605 5274 cpu_manager.go:414] "containerMap: removing container" podUID="f42a5beb-ad0f-43fc-97ad-0af6febd68ee" containerName="thanos-sidecar" Oct 31 00:50:20.034038 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.033609 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f42a5beb-ad0f-43fc-97ad-0af6febd68ee" containerName="thanos-sidecar" Oct 31 00:50:20.034038 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.033619 5274 cpu_manager.go:414] "containerMap: removing container" podUID="149823e9-b0e4-476c-9d37-d4581e7ea7d1" containerName="kube-rbac-proxy" Oct 31 00:50:20.034038 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.033624 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="149823e9-b0e4-476c-9d37-d4581e7ea7d1" containerName="kube-rbac-proxy" Oct 31 00:50:20.034038 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.033631 5274 cpu_manager.go:414] "containerMap: removing container" podUID="9777d542-77d4-4ad3-b2c5-a60b8ec24d68" containerName="config-reloader" Oct 31 00:50:20.034038 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.033636 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9777d542-77d4-4ad3-b2c5-a60b8ec24d68" containerName="config-reloader" Oct 31 00:50:20.034038 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.033642 5274 cpu_manager.go:414] "containerMap: removing container" podUID="976644b4-136a-42bd-95c8-c5dd1e415ac6" containerName="otc-container" Oct 31 00:50:20.034038 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.033646 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="976644b4-136a-42bd-95c8-c5dd1e415ac6" containerName="otc-container" Oct 31 00:50:20.034038 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.033690 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="f42a5beb-ad0f-43fc-97ad-0af6febd68ee" containerName="config-reloader" Oct 31 00:50:20.034038 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.033699 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="f42a5beb-ad0f-43fc-97ad-0af6febd68ee" containerName="prometheus" Oct 31 00:50:20.034038 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.033706 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="58d39fb2-6f91-4f59-aa7c-45e69e43cb12" containerName="korrel8r" Oct 31 00:50:20.034038 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.033711 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="9777d542-77d4-4ad3-b2c5-a60b8ec24d68" containerName="alertmanager" Oct 31 00:50:20.034038 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.033717 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="f42a5beb-ad0f-43fc-97ad-0af6febd68ee" containerName="thanos-sidecar" Oct 31 00:50:20.034038 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.033723 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="149823e9-b0e4-476c-9d37-d4581e7ea7d1" containerName="kube-rbac-proxy" Oct 31 00:50:20.034038 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.033730 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="9777d542-77d4-4ad3-b2c5-a60b8ec24d68" containerName="config-reloader" Oct 31 00:50:20.034038 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.033734 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="976644b4-136a-42bd-95c8-c5dd1e415ac6" containerName="otc-container" Oct 31 00:50:20.038618 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.038599 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-fmzfj" Oct 31 00:50:20.048984 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.048947 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:50:20.050359 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.050331 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:50:20.050470 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.050453 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:50:20.050540 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.050519 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\"" Oct 31 00:50:20.051404 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.051384 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 31 00:50:20.051503 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.051403 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\"" Oct 31 00:50:20.051503 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.051442 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-qr24b\"" Oct 31 00:50:20.052105 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.052088 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-fmzfj"] Oct 31 00:50:20.108086 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.108062 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-8gsx4"] Oct 31 00:50:20.111343 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.111328 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8gsx4" Oct 31 00:50:20.121449 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.121427 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-8gsx4"] Oct 31 00:50:20.124753 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.124731 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-6q4d4\"" Oct 31 00:50:20.124861 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.124844 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 31 00:50:20.139775 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.139750 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/6131e5be-2a37-463e-ad2b-d1f60b831a7e-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-fmzfj\" (UID: \"6131e5be-2a37-463e-ad2b-d1f60b831a7e\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-fmzfj" Oct 31 00:50:20.139865 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.139789 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/6131e5be-2a37-463e-ad2b-d1f60b831a7e-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-fmzfj\" (UID: \"6131e5be-2a37-463e-ad2b-d1f60b831a7e\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-fmzfj" Oct 31 00:50:20.139865 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.139807 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7spdk\" (UniqueName: \"kubernetes.io/projected/6131e5be-2a37-463e-ad2b-d1f60b831a7e-kube-api-access-7spdk\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-fmzfj\" (UID: \"6131e5be-2a37-463e-ad2b-d1f60b831a7e\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-fmzfj" Oct 31 00:50:20.139938 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.139926 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6131e5be-2a37-463e-ad2b-d1f60b831a7e-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-fmzfj\" (UID: \"6131e5be-2a37-463e-ad2b-d1f60b831a7e\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-fmzfj" Oct 31 00:50:20.139992 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.139971 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6131e5be-2a37-463e-ad2b-d1f60b831a7e-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-fmzfj\" (UID: \"6131e5be-2a37-463e-ad2b-d1f60b831a7e\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-fmzfj" Oct 31 00:50:20.148903 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.148885 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 31 00:50:20.240889 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.240870 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/6131e5be-2a37-463e-ad2b-d1f60b831a7e-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-fmzfj\" (UID: \"6131e5be-2a37-463e-ad2b-d1f60b831a7e\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-fmzfj" Oct 31 00:50:20.241034 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.240908 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c56cc3d3-d13b-4b98-a94f-4de0d7d6f796-config\") pod \"korrel8r-5cbcc89b8f-8gsx4\" (UID: \"c56cc3d3-d13b-4b98-a94f-4de0d7d6f796\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8gsx4" Oct 31 00:50:20.241034 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.240926 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/6131e5be-2a37-463e-ad2b-d1f60b831a7e-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-fmzfj\" (UID: \"6131e5be-2a37-463e-ad2b-d1f60b831a7e\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-fmzfj" Oct 31 00:50:20.241034 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.240947 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7spdk\" (UniqueName: \"kubernetes.io/projected/6131e5be-2a37-463e-ad2b-d1f60b831a7e-kube-api-access-7spdk\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-fmzfj\" (UID: \"6131e5be-2a37-463e-ad2b-d1f60b831a7e\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-fmzfj" Oct 31 00:50:20.241034 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:50:20.241029 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found Oct 31 00:50:20.241316 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.241034 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6131e5be-2a37-463e-ad2b-d1f60b831a7e-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-fmzfj\" (UID: \"6131e5be-2a37-463e-ad2b-d1f60b831a7e\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-fmzfj" Oct 31 00:50:20.241316 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.241070 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6131e5be-2a37-463e-ad2b-d1f60b831a7e-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-fmzfj\" (UID: \"6131e5be-2a37-463e-ad2b-d1f60b831a7e\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-fmzfj" Oct 31 00:50:20.241316 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:50:20.241119 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6131e5be-2a37-463e-ad2b-d1f60b831a7e-prometheus-client-cert podName:6131e5be-2a37-463e-ad2b-d1f60b831a7e nodeName:}" failed. No retries permitted until 2026-10-31 00:50:20.741069574 +0000 UTC m=+3057.377888289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/6131e5be-2a37-463e-ad2b-d1f60b831a7e-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-fmzfj" (UID: "6131e5be-2a37-463e-ad2b-d1f60b831a7e") : secret "prometheus-operated-tls" not found Oct 31 00:50:20.241316 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.241161 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knngm\" (UniqueName: \"kubernetes.io/projected/c56cc3d3-d13b-4b98-a94f-4de0d7d6f796-kube-api-access-knngm\") pod \"korrel8r-5cbcc89b8f-8gsx4\" (UID: \"c56cc3d3-d13b-4b98-a94f-4de0d7d6f796\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8gsx4" Oct 31 00:50:20.241316 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.241236 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c56cc3d3-d13b-4b98-a94f-4de0d7d6f796-tls\") pod \"korrel8r-5cbcc89b8f-8gsx4\" (UID: \"c56cc3d3-d13b-4b98-a94f-4de0d7d6f796\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8gsx4" Oct 31 00:50:20.241950 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.241930 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/6131e5be-2a37-463e-ad2b-d1f60b831a7e-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-fmzfj\" (UID: \"6131e5be-2a37-463e-ad2b-d1f60b831a7e\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-fmzfj" Oct 31 00:50:20.244314 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.244293 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6131e5be-2a37-463e-ad2b-d1f60b831a7e-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-fmzfj\" (UID: \"6131e5be-2a37-463e-ad2b-d1f60b831a7e\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-fmzfj" Oct 31 00:50:20.244663 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.244644 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6131e5be-2a37-463e-ad2b-d1f60b831a7e-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-fmzfj\" (UID: \"6131e5be-2a37-463e-ad2b-d1f60b831a7e\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-fmzfj" Oct 31 00:50:20.258299 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.258273 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7spdk\" (UniqueName: \"kubernetes.io/projected/6131e5be-2a37-463e-ad2b-d1f60b831a7e-kube-api-access-7spdk\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-fmzfj\" (UID: \"6131e5be-2a37-463e-ad2b-d1f60b831a7e\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-fmzfj" Oct 31 00:50:20.342088 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.341999 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knngm\" (UniqueName: \"kubernetes.io/projected/c56cc3d3-d13b-4b98-a94f-4de0d7d6f796-kube-api-access-knngm\") pod \"korrel8r-5cbcc89b8f-8gsx4\" (UID: \"c56cc3d3-d13b-4b98-a94f-4de0d7d6f796\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8gsx4" Oct 31 00:50:20.342088 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.342070 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c56cc3d3-d13b-4b98-a94f-4de0d7d6f796-tls\") pod \"korrel8r-5cbcc89b8f-8gsx4\" (UID: \"c56cc3d3-d13b-4b98-a94f-4de0d7d6f796\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8gsx4" Oct 31 00:50:20.342305 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.342123 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c56cc3d3-d13b-4b98-a94f-4de0d7d6f796-config\") pod \"korrel8r-5cbcc89b8f-8gsx4\" (UID: \"c56cc3d3-d13b-4b98-a94f-4de0d7d6f796\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8gsx4" Oct 31 00:50:20.342305 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:50:20.342166 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/korrel8r-tls: secret "korrel8r-tls" not found Oct 31 00:50:20.342305 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:50:20.342240 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c56cc3d3-d13b-4b98-a94f-4de0d7d6f796-tls podName:c56cc3d3-d13b-4b98-a94f-4de0d7d6f796 nodeName:}" failed. No retries permitted until 2026-10-31 00:50:20.842225413 +0000 UTC m=+3057.479044133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls" (UniqueName: "kubernetes.io/secret/c56cc3d3-d13b-4b98-a94f-4de0d7d6f796-tls") pod "korrel8r-5cbcc89b8f-8gsx4" (UID: "c56cc3d3-d13b-4b98-a94f-4de0d7d6f796") : secret "korrel8r-tls" not found Oct 31 00:50:20.342854 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.342837 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c56cc3d3-d13b-4b98-a94f-4de0d7d6f796-config\") pod \"korrel8r-5cbcc89b8f-8gsx4\" (UID: \"c56cc3d3-d13b-4b98-a94f-4de0d7d6f796\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8gsx4" Oct 31 00:50:20.357411 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.357387 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knngm\" (UniqueName: \"kubernetes.io/projected/c56cc3d3-d13b-4b98-a94f-4de0d7d6f796-kube-api-access-knngm\") pod \"korrel8r-5cbcc89b8f-8gsx4\" (UID: \"c56cc3d3-d13b-4b98-a94f-4de0d7d6f796\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8gsx4" Oct 31 00:50:20.579130 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.579094 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:50:20.583626 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.583602 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:20.591437 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.591399 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:50:20.592060 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.592035 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:50:20.592182 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.592152 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:50:20.592231 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.592209 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:50:20.592879 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.592863 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:50:20.596527 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.596505 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:50:20.645244 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.645208 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db1066ed-101f-436d-916e-88fea4540e87-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"db1066ed-101f-436d-916e-88fea4540e87\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:20.645415 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.645277 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/db1066ed-101f-436d-916e-88fea4540e87-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"db1066ed-101f-436d-916e-88fea4540e87\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:20.645415 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.645355 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/db1066ed-101f-436d-916e-88fea4540e87-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"db1066ed-101f-436d-916e-88fea4540e87\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:20.645415 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.645407 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/db1066ed-101f-436d-916e-88fea4540e87-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"db1066ed-101f-436d-916e-88fea4540e87\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:20.645597 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.645541 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msgf5\" (UniqueName: \"kubernetes.io/projected/db1066ed-101f-436d-916e-88fea4540e87-kube-api-access-msgf5\") pod \"data-science-collector-collector-0\" (UID: \"db1066ed-101f-436d-916e-88fea4540e87\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:20.645653 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.645635 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/db1066ed-101f-436d-916e-88fea4540e87-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"db1066ed-101f-436d-916e-88fea4540e87\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:20.746297 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.746267 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/db1066ed-101f-436d-916e-88fea4540e87-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"db1066ed-101f-436d-916e-88fea4540e87\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:20.746442 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.746306 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/db1066ed-101f-436d-916e-88fea4540e87-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"db1066ed-101f-436d-916e-88fea4540e87\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:20.746442 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.746334 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msgf5\" (UniqueName: \"kubernetes.io/projected/db1066ed-101f-436d-916e-88fea4540e87-kube-api-access-msgf5\") pod \"data-science-collector-collector-0\" (UID: \"db1066ed-101f-436d-916e-88fea4540e87\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:20.746442 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.746389 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/db1066ed-101f-436d-916e-88fea4540e87-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"db1066ed-101f-436d-916e-88fea4540e87\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:20.746442 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.746418 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/6131e5be-2a37-463e-ad2b-d1f60b831a7e-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-fmzfj\" (UID: \"6131e5be-2a37-463e-ad2b-d1f60b831a7e\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-fmzfj" Oct 31 00:50:20.746688 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.746446 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db1066ed-101f-436d-916e-88fea4540e87-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"db1066ed-101f-436d-916e-88fea4540e87\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:20.746688 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.746489 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/db1066ed-101f-436d-916e-88fea4540e87-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"db1066ed-101f-436d-916e-88fea4540e87\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:20.746688 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:50:20.746518 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found Oct 31 00:50:20.746688 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:50:20.746576 5274 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found Oct 31 00:50:20.746688 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:50:20.746590 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6131e5be-2a37-463e-ad2b-d1f60b831a7e-prometheus-client-cert podName:6131e5be-2a37-463e-ad2b-d1f60b831a7e nodeName:}" failed. No retries permitted until 2026-10-31 00:50:21.746565848 +0000 UTC m=+3058.383384560 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/6131e5be-2a37-463e-ad2b-d1f60b831a7e-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-fmzfj" (UID: "6131e5be-2a37-463e-ad2b-d1f60b831a7e") : secret "prometheus-operated-tls" not found Oct 31 00:50:20.746688 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:50:20.746623 5274 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db1066ed-101f-436d-916e-88fea4540e87-otlp-tls-certs podName:db1066ed-101f-436d-916e-88fea4540e87 nodeName:}" failed. No retries permitted until 2026-10-31 00:50:21.24660716 +0000 UTC m=+3057.883425868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/db1066ed-101f-436d-916e-88fea4540e87-otlp-tls-certs") pod "data-science-collector-collector-0" (UID: "db1066ed-101f-436d-916e-88fea4540e87") : secret "data-science-collector-collector-headless-tls" not found Oct 31 00:50:20.747155 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.747129 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/db1066ed-101f-436d-916e-88fea4540e87-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"db1066ed-101f-436d-916e-88fea4540e87\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:20.749041 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.749019 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db1066ed-101f-436d-916e-88fea4540e87-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"db1066ed-101f-436d-916e-88fea4540e87\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:20.749723 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.749407 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/db1066ed-101f-436d-916e-88fea4540e87-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"db1066ed-101f-436d-916e-88fea4540e87\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:20.749723 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.749415 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/db1066ed-101f-436d-916e-88fea4540e87-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"db1066ed-101f-436d-916e-88fea4540e87\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:20.758791 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.758772 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msgf5\" (UniqueName: \"kubernetes.io/projected/db1066ed-101f-436d-916e-88fea4540e87-kube-api-access-msgf5\") pod \"data-science-collector-collector-0\" (UID: \"db1066ed-101f-436d-916e-88fea4540e87\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:20.847159 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.847103 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c56cc3d3-d13b-4b98-a94f-4de0d7d6f796-tls\") pod \"korrel8r-5cbcc89b8f-8gsx4\" (UID: \"c56cc3d3-d13b-4b98-a94f-4de0d7d6f796\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8gsx4" Oct 31 00:50:20.849179 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:20.849159 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c56cc3d3-d13b-4b98-a94f-4de0d7d6f796-tls\") pod \"korrel8r-5cbcc89b8f-8gsx4\" (UID: \"c56cc3d3-d13b-4b98-a94f-4de0d7d6f796\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8gsx4" Oct 31 00:50:20.899193 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:50:20.899161 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:50:21.024202 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:21.024178 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8gsx4" Oct 31 00:50:21.250435 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:21.250409 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/db1066ed-101f-436d-916e-88fea4540e87-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"db1066ed-101f-436d-916e-88fea4540e87\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:21.252876 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:21.252851 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/db1066ed-101f-436d-916e-88fea4540e87-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"db1066ed-101f-436d-916e-88fea4540e87\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:21.299348 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:21.299276 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-8gsx4"] Oct 31 00:50:21.301529 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:50:21.301507 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc56cc3d3_d13b_4b98_a94f_4de0d7d6f796.slice/crio-89107d1e64c4ad045f31135f20a2bee7a7988e7a6c80b590a61600f80d5f144c WatchSource:0}: Error finding container 89107d1e64c4ad045f31135f20a2bee7a7988e7a6c80b590a61600f80d5f144c: Status 404 returned error can't find the container with id 89107d1e64c4ad045f31135f20a2bee7a7988e7a6c80b590a61600f80d5f144c Oct 31 00:50:21.324421 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:21.324398 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8gsx4" event={"ID":"c56cc3d3-d13b-4b98-a94f-4de0d7d6f796","Type":"ContainerStarted","Data":"89107d1e64c4ad045f31135f20a2bee7a7988e7a6c80b590a61600f80d5f144c"} Oct 31 00:50:21.506223 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:21.506156 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:21.754641 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:21.754610 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/6131e5be-2a37-463e-ad2b-d1f60b831a7e-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-fmzfj\" (UID: \"6131e5be-2a37-463e-ad2b-d1f60b831a7e\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-fmzfj" Oct 31 00:50:21.756983 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:21.756912 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/6131e5be-2a37-463e-ad2b-d1f60b831a7e-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-fmzfj\" (UID: \"6131e5be-2a37-463e-ad2b-d1f60b831a7e\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-fmzfj" Oct 31 00:50:21.782798 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:21.782759 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:50:21.789291 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:50:21.789267 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb1066ed_101f_436d_916e_88fea4540e87.slice/crio-9eb6222cccc10ba23a4537fe5732f252a9555ae874a32e128b3f893b26304237 WatchSource:0}: Error finding container 9eb6222cccc10ba23a4537fe5732f252a9555ae874a32e128b3f893b26304237: Status 404 returned error can't find the container with id 9eb6222cccc10ba23a4537fe5732f252a9555ae874a32e128b3f893b26304237 Oct 31 00:50:21.791360 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:21.791329 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:50:21.791438 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:21.791382 5274 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 00:50:21.852085 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:21.852065 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-fmzfj" Oct 31 00:50:22.125392 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:22.125182 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-fmzfj"] Oct 31 00:50:22.127807 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:50:22.127781 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6131e5be_2a37_463e_ad2b_d1f60b831a7e.slice/crio-26340255553df18cc716fa1dcd899090e9fe54b6270d56cd95a571c170f0f3ba WatchSource:0}: Error finding container 26340255553df18cc716fa1dcd899090e9fe54b6270d56cd95a571c170f0f3ba: Status 404 returned error can't find the container with id 26340255553df18cc716fa1dcd899090e9fe54b6270d56cd95a571c170f0f3ba Oct 31 00:50:22.331326 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:22.331294 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8gsx4" event={"ID":"c56cc3d3-d13b-4b98-a94f-4de0d7d6f796","Type":"ContainerStarted","Data":"15ad16eb9092c758f9e004a6963c0ac0938dd4dc623cee3695daf996810cba15"} Oct 31 00:50:22.331725 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:22.331406 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8gsx4" Oct 31 00:50:22.332971 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:22.332936 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"db1066ed-101f-436d-916e-88fea4540e87","Type":"ContainerStarted","Data":"26e8386dc53d861ae4a79be0d6da7cca684f6e4a95dbbac3b63ba4472cab1109"} Oct 31 00:50:22.333103 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:22.332987 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"db1066ed-101f-436d-916e-88fea4540e87","Type":"ContainerStarted","Data":"9eb6222cccc10ba23a4537fe5732f252a9555ae874a32e128b3f893b26304237"} Oct 31 00:50:22.334171 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:22.334148 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-fmzfj" event={"ID":"6131e5be-2a37-463e-ad2b-d1f60b831a7e","Type":"ContainerStarted","Data":"26340255553df18cc716fa1dcd899090e9fe54b6270d56cd95a571c170f0f3ba"} Oct 31 00:50:22.351590 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:22.351549 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8gsx4" podStartSLOduration=2.351533161 podStartE2EDuration="2.351533161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:22.350950324 +0000 UTC m=+3058.987769057" watchObservedRunningTime="2026-10-31 00:50:22.351533161 +0000 UTC m=+3058.988351877" Oct 31 00:50:22.373196 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:22.373153 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.373141509 podStartE2EDuration="2.373141509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:22.371130377 +0000 UTC m=+3059.007949109" watchObservedRunningTime="2026-10-31 00:50:22.373141509 +0000 UTC m=+3059.009960222" Oct 31 00:50:23.341283 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:23.341243 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-fmzfj" event={"ID":"6131e5be-2a37-463e-ad2b-d1f60b831a7e","Type":"ContainerStarted","Data":"e4631160363c736227355b173232b8346029c508918585168f349944c235a53f"} Oct 31 00:50:23.341669 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:23.341319 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-fmzfj" Oct 31 00:50:26.563539 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.563470 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-fmzfj" podStartSLOduration=6.563451403 podStartE2EDuration="6.563451403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:23.361287734 +0000 UTC m=+3059.998106449" watchObservedRunningTime="2026-10-31 00:50:26.563451403 +0000 UTC m=+3063.200270120" Oct 31 00:50:26.563946 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.563781 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:50:26.568261 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.568239 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:50:26.572254 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.572200 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 31 00:50:26.572254 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.572226 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 31 00:50:26.572408 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.572270 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:50:26.572408 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.572308 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-rdq4v\"" Oct 31 00:50:26.573097 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.573083 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 31 00:50:26.587328 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.587275 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:50:26.697562 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.697537 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6cea6fd7-14ad-45ce-b408-6042192c0a97-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"6cea6fd7-14ad-45ce-b408-6042192c0a97\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:50:26.697690 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.697568 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wbr6\" (UniqueName: \"kubernetes.io/projected/6cea6fd7-14ad-45ce-b408-6042192c0a97-kube-api-access-8wbr6\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"6cea6fd7-14ad-45ce-b408-6042192c0a97\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:50:26.697690 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.697600 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/6cea6fd7-14ad-45ce-b408-6042192c0a97-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"6cea6fd7-14ad-45ce-b408-6042192c0a97\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:50:26.697800 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.697686 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6cea6fd7-14ad-45ce-b408-6042192c0a97-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"6cea6fd7-14ad-45ce-b408-6042192c0a97\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:50:26.697800 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.697791 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6cea6fd7-14ad-45ce-b408-6042192c0a97-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"6cea6fd7-14ad-45ce-b408-6042192c0a97\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:50:26.697876 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.697812 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6cea6fd7-14ad-45ce-b408-6042192c0a97-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"6cea6fd7-14ad-45ce-b408-6042192c0a97\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:50:26.697876 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.697831 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6cea6fd7-14ad-45ce-b408-6042192c0a97-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"6cea6fd7-14ad-45ce-b408-6042192c0a97\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:50:26.798972 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.798934 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6cea6fd7-14ad-45ce-b408-6042192c0a97-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"6cea6fd7-14ad-45ce-b408-6042192c0a97\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:50:26.799121 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.799007 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6cea6fd7-14ad-45ce-b408-6042192c0a97-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"6cea6fd7-14ad-45ce-b408-6042192c0a97\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:50:26.799121 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.799102 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wbr6\" (UniqueName: \"kubernetes.io/projected/6cea6fd7-14ad-45ce-b408-6042192c0a97-kube-api-access-8wbr6\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"6cea6fd7-14ad-45ce-b408-6042192c0a97\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:50:26.799206 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.799161 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/6cea6fd7-14ad-45ce-b408-6042192c0a97-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"6cea6fd7-14ad-45ce-b408-6042192c0a97\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:50:26.799266 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.799206 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6cea6fd7-14ad-45ce-b408-6042192c0a97-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"6cea6fd7-14ad-45ce-b408-6042192c0a97\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:50:26.799326 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.799277 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6cea6fd7-14ad-45ce-b408-6042192c0a97-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"6cea6fd7-14ad-45ce-b408-6042192c0a97\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:50:26.799326 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.799304 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6cea6fd7-14ad-45ce-b408-6042192c0a97-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"6cea6fd7-14ad-45ce-b408-6042192c0a97\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:50:26.799584 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.799557 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/6cea6fd7-14ad-45ce-b408-6042192c0a97-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"6cea6fd7-14ad-45ce-b408-6042192c0a97\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:50:26.801311 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.801287 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6cea6fd7-14ad-45ce-b408-6042192c0a97-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"6cea6fd7-14ad-45ce-b408-6042192c0a97\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:50:26.801409 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.801321 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6cea6fd7-14ad-45ce-b408-6042192c0a97-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"6cea6fd7-14ad-45ce-b408-6042192c0a97\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:50:26.801512 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.801495 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6cea6fd7-14ad-45ce-b408-6042192c0a97-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"6cea6fd7-14ad-45ce-b408-6042192c0a97\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:50:26.801726 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.801709 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6cea6fd7-14ad-45ce-b408-6042192c0a97-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"6cea6fd7-14ad-45ce-b408-6042192c0a97\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:50:26.801763 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.801731 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6cea6fd7-14ad-45ce-b408-6042192c0a97-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"6cea6fd7-14ad-45ce-b408-6042192c0a97\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:50:26.812161 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.812140 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wbr6\" (UniqueName: \"kubernetes.io/projected/6cea6fd7-14ad-45ce-b408-6042192c0a97-kube-api-access-8wbr6\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"6cea6fd7-14ad-45ce-b408-6042192c0a97\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:50:26.851340 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.851284 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:50:26.855282 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.855263 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:26.871573 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.871543 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:50:26.888332 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.888308 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:50:26.940097 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.939314 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 31 00:50:26.940097 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.939466 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-zqm6n\"" Oct 31 00:50:26.940097 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.939580 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 31 00:50:26.940097 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.939645 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 31 00:50:26.940097 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.939745 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:50:26.940097 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.939755 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 31 00:50:26.940097 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.939817 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:50:26.940097 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.939811 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 31 00:50:26.940097 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:26.939919 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 31 00:50:27.000631 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:27.000605 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:27.000778 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:27.000638 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:27.000778 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:27.000672 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:27.000778 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:27.000750 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:27.000948 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:27.000815 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:27.000948 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:27.000855 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:27.000948 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:27.000886 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:27.001095 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:27.001017 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:27.001095 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:27.001048 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:27.001095 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:27.001070 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm592\" (UniqueName: \"kubernetes.io/projected/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-kube-api-access-mm592\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:27.001243 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:27.001122 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:27.001243 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:27.001190 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:27.101627 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:27.101570 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:27.101627 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:27.101600 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:27.101627 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:27.101623 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:27.101851 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:27.101647 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:27.101851 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:27.101821 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:27.101947 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:27.101887 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:27.101947 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:27.101919 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:27.102139 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:27.101947 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:27.102139 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:27.102004 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:27.102139 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:27.102097 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:27.102139 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:27.102135 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:27.102347 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:27.102162 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm592\" (UniqueName: \"kubernetes.io/projected/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-kube-api-access-mm592\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:27.102608 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:27.102576 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:27.103633 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:27.103314 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:27.103633 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:27.103457 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:27.104690 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:27.104665 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:27.104782 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:27.104756 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:27.104833 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:27.104756 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:27.104833 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:27.104792 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:27.105156 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:27.105140 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:27.105242 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:27.105221 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:27.105679 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:27.105659 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:27.106041 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:27.105998 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:27.114231 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:27.114208 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm592\" (UniqueName: \"kubernetes.io/projected/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-kube-api-access-mm592\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:27.168622 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:27.168597 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:50:27.171337 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:50:27.171308 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cea6fd7_14ad_45ce_b408_6042192c0a97.slice/crio-c189e0f3ce893a40d783015255c174218c89e282dae752fd9e9866489a11c82a WatchSource:0}: Error finding container c189e0f3ce893a40d783015255c174218c89e282dae752fd9e9866489a11c82a: Status 404 returned error can't find the container with id c189e0f3ce893a40d783015255c174218c89e282dae752fd9e9866489a11c82a Oct 31 00:50:27.361787 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:27.361722 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"6cea6fd7-14ad-45ce-b408-6042192c0a97","Type":"ContainerStarted","Data":"c189e0f3ce893a40d783015255c174218c89e282dae752fd9e9866489a11c82a"} Oct 31 00:50:28.368579 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:28.368549 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"6cea6fd7-14ad-45ce-b408-6042192c0a97","Type":"ContainerStarted","Data":"0b0c8265438d78f5dc0cd70a79231a3c92da53c6b6a03b3199811126e448a055"} Oct 31 00:50:29.348686 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:29.348660 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-fmzfj" Oct 31 00:50:31.385716 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:31.385678 5274 generic.go:356] "Generic (PLEG): container finished" podID="6cea6fd7-14ad-45ce-b408-6042192c0a97" containerID="0b0c8265438d78f5dc0cd70a79231a3c92da53c6b6a03b3199811126e448a055" exitCode=0 Oct 31 00:50:31.386119 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:31.385745 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"6cea6fd7-14ad-45ce-b408-6042192c0a97","Type":"ContainerDied","Data":"0b0c8265438d78f5dc0cd70a79231a3c92da53c6b6a03b3199811126e448a055"} Oct 31 00:50:32.393223 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:32.393192 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"6cea6fd7-14ad-45ce-b408-6042192c0a97","Type":"ContainerStarted","Data":"cd9d621ef5db0864be3004de60359a02ccfed1c46dd7b073fbb26e9a2df1f6e2"} Oct 31 00:50:32.393661 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:32.393260 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"6cea6fd7-14ad-45ce-b408-6042192c0a97","Type":"ContainerStarted","Data":"70368cbfb832babccdf637b2863839a4daad2e3fae1a73052f16c011c3482629"} Oct 31 00:50:32.393661 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:32.393473 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:50:32.422781 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:32.422729 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=3.337077243 podStartE2EDuration="6.422716471s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.085639228s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:32.420595072 +0000 UTC m=+3069.057413998" watchObservedRunningTime="2026-10-31 00:50:32.422716471 +0000 UTC m=+3069.059535187" Oct 31 00:50:33.343693 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:33.343665 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8gsx4" Oct 31 00:50:35.372880 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:35.372848 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:35.380204 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:35.380178 5274 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:35.474171 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:35.474145 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:35.474283 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:35.474268 5274 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:35.477929 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:35.477906 5274 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") DevicePath \"csi-3beb91b3ebf177571ca474589da3560cd18787cfa50b2a0b9bfef85661e6cf21\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:35.568840 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:35.568818 5274 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/534eb232558c8505fa55558aa87cc96e2d2b0adb946edb717939ec387be54e97/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:35.584809 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:35.584788 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:35.875921 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:35.875895 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:36.166048 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:36.166024 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:50:36.167014 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:50:36.166986 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod094e5b88_2f67_4ec4_9847_f03dfd7eca5b.slice/crio-56534a8d5c663fff8a6e75f374a07d555ce7650e5e9579fe793c9dd101844fc3 WatchSource:0}: Error finding container 56534a8d5c663fff8a6e75f374a07d555ce7650e5e9579fe793c9dd101844fc3: Status 404 returned error can't find the container with id 56534a8d5c663fff8a6e75f374a07d555ce7650e5e9579fe793c9dd101844fc3 Oct 31 00:50:36.413472 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:36.413442 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"094e5b88-2f67-4ec4-9847-f03dfd7eca5b","Type":"ContainerStarted","Data":"56534a8d5c663fff8a6e75f374a07d555ce7650e5e9579fe793c9dd101844fc3"} Oct 31 00:50:37.425985 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:37.425925 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"094e5b88-2f67-4ec4-9847-f03dfd7eca5b","Type":"ContainerStarted","Data":"d2c7e3e8613de4cdc5fcc833b8ec145e800ae8a809e5506df0ad5077f9477e3e"} Oct 31 00:50:38.399626 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:38.399598 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:50:40.444301 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:40.444266 5274 generic.go:356] "Generic (PLEG): container finished" podID="094e5b88-2f67-4ec4-9847-f03dfd7eca5b" containerID="d2c7e3e8613de4cdc5fcc833b8ec145e800ae8a809e5506df0ad5077f9477e3e" exitCode=0 Oct 31 00:50:40.444770 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:40.444339 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"094e5b88-2f67-4ec4-9847-f03dfd7eca5b","Type":"ContainerDied","Data":"d2c7e3e8613de4cdc5fcc833b8ec145e800ae8a809e5506df0ad5077f9477e3e"} Oct 31 00:50:41.452884 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:41.452853 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"094e5b88-2f67-4ec4-9847-f03dfd7eca5b","Type":"ContainerStarted","Data":"e6eb5563f443744d815dd7b6126708784884f4ff4e89d043adf39cc487342adc"} Oct 31 00:50:41.453376 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:41.452915 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"094e5b88-2f67-4ec4-9847-f03dfd7eca5b","Type":"ContainerStarted","Data":"6f46ceff18145e6db5f1dc29192fadcd43c3bd795aba262dac20a290d1d669e6"} Oct 31 00:50:41.453376 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:41.452943 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"094e5b88-2f67-4ec4-9847-f03dfd7eca5b","Type":"ContainerStarted","Data":"ad42ac969a0002b6039492ed99ef0f6913c3a58ee1153a720d98d32ae4b58ff5"} Oct 31 00:50:41.489877 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:41.489815 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=12.303239917 podStartE2EDuration="15.489798164s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.186558247s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:41.484790767 +0000 UTC m=+3078.121609484" watchObservedRunningTime="2026-10-31 00:50:41.489798164 +0000 UTC m=+3078.126616880" Oct 31 00:50:45.876984 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:45.876937 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:50.877158 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:50.877120 5274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:50.882625 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:50.882599 5274 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:50.900322 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:50:50.900296 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:50:51.507649 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:51.507619 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:53.717295 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:53.717259 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-fmzfj"] Oct 31 00:50:53.717701 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:53.717450 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-fmzfj" podUID="6131e5be-2a37-463e-ad2b-d1f60b831a7e" containerName="kube-rbac-proxy" containerID="cri-o://e4631160363c736227355b173232b8346029c508918585168f349944c235a53f" gracePeriod=30 Oct 31 00:50:53.734246 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:53.734216 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-8gsx4"] Oct 31 00:50:53.734398 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:53.734379 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8gsx4" podUID="c56cc3d3-d13b-4b98-a94f-4de0d7d6f796" containerName="korrel8r" containerID="cri-o://15ad16eb9092c758f9e004a6963c0ac0938dd4dc623cee3695daf996810cba15" gracePeriod=30 Oct 31 00:50:53.941595 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:53.938638 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:50:53.941595 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:53.939215 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="6cea6fd7-14ad-45ce-b408-6042192c0a97" containerName="alertmanager" containerID="cri-o://70368cbfb832babccdf637b2863839a4daad2e3fae1a73052f16c011c3482629" gracePeriod=120 Oct 31 00:50:53.941595 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:53.939349 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="6cea6fd7-14ad-45ce-b408-6042192c0a97" containerName="config-reloader" containerID="cri-o://cd9d621ef5db0864be3004de60359a02ccfed1c46dd7b073fbb26e9a2df1f6e2" gracePeriod=120 Oct 31 00:50:53.955362 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:53.955203 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:50:53.956375 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:53.955809 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="094e5b88-2f67-4ec4-9847-f03dfd7eca5b" containerName="prometheus" containerID="cri-o://ad42ac969a0002b6039492ed99ef0f6913c3a58ee1153a720d98d32ae4b58ff5" gracePeriod=600 Oct 31 00:50:53.956375 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:53.956030 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="094e5b88-2f67-4ec4-9847-f03dfd7eca5b" containerName="thanos-sidecar" containerID="cri-o://e6eb5563f443744d815dd7b6126708784884f4ff4e89d043adf39cc487342adc" gracePeriod=600 Oct 31 00:50:53.956375 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:53.956109 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="094e5b88-2f67-4ec4-9847-f03dfd7eca5b" containerName="config-reloader" containerID="cri-o://6f46ceff18145e6db5f1dc29192fadcd43c3bd795aba262dac20a290d1d669e6" gracePeriod=600 Oct 31 00:50:54.084845 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.084711 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-fmzfj" Oct 31 00:50:54.088153 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.088105 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8gsx4" Oct 31 00:50:54.124973 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.111446 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:50:54.124973 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.111764 5274 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="db1066ed-101f-436d-916e-88fea4540e87" containerName="otc-container" containerID="cri-o://26e8386dc53d861ae4a79be0d6da7cca684f6e4a95dbbac3b63ba4472cab1109" gracePeriod=30 Oct 31 00:50:54.127902 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.127875 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6131e5be-2a37-463e-ad2b-d1f60b831a7e-tls\" (UniqueName: \"kubernetes.io/secret/6131e5be-2a37-463e-ad2b-d1f60b831a7e-tls\") pod \"6131e5be-2a37-463e-ad2b-d1f60b831a7e\" (UID: \"6131e5be-2a37-463e-ad2b-d1f60b831a7e\") " Oct 31 00:50:54.128062 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.127937 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c56cc3d3-d13b-4b98-a94f-4de0d7d6f796-config\" (UniqueName: \"kubernetes.io/configmap/c56cc3d3-d13b-4b98-a94f-4de0d7d6f796-config\") pod \"c56cc3d3-d13b-4b98-a94f-4de0d7d6f796\" (UID: \"c56cc3d3-d13b-4b98-a94f-4de0d7d6f796\") " Oct 31 00:50:54.128062 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.127979 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6131e5be-2a37-463e-ad2b-d1f60b831a7e-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/6131e5be-2a37-463e-ad2b-d1f60b831a7e-prometheus-ca\") pod \"6131e5be-2a37-463e-ad2b-d1f60b831a7e\" (UID: \"6131e5be-2a37-463e-ad2b-d1f60b831a7e\") " Oct 31 00:50:54.128062 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.128012 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6131e5be-2a37-463e-ad2b-d1f60b831a7e-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/6131e5be-2a37-463e-ad2b-d1f60b831a7e-prometheus-client-cert\") pod \"6131e5be-2a37-463e-ad2b-d1f60b831a7e\" (UID: \"6131e5be-2a37-463e-ad2b-d1f60b831a7e\") " Oct 31 00:50:54.128062 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.128042 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6131e5be-2a37-463e-ad2b-d1f60b831a7e-kube-api-access-7spdk\" (UniqueName: \"kubernetes.io/projected/6131e5be-2a37-463e-ad2b-d1f60b831a7e-kube-api-access-7spdk\") pod \"6131e5be-2a37-463e-ad2b-d1f60b831a7e\" (UID: \"6131e5be-2a37-463e-ad2b-d1f60b831a7e\") " Oct 31 00:50:54.128062 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.128060 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c56cc3d3-d13b-4b98-a94f-4de0d7d6f796-tls\" (UniqueName: \"kubernetes.io/secret/c56cc3d3-d13b-4b98-a94f-4de0d7d6f796-tls\") pod \"c56cc3d3-d13b-4b98-a94f-4de0d7d6f796\" (UID: \"c56cc3d3-d13b-4b98-a94f-4de0d7d6f796\") " Oct 31 00:50:54.128355 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.128103 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6131e5be-2a37-463e-ad2b-d1f60b831a7e-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6131e5be-2a37-463e-ad2b-d1f60b831a7e-kube-rbac-proxy-config\") pod \"6131e5be-2a37-463e-ad2b-d1f60b831a7e\" (UID: \"6131e5be-2a37-463e-ad2b-d1f60b831a7e\") " Oct 31 00:50:54.128355 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.128165 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c56cc3d3-d13b-4b98-a94f-4de0d7d6f796-kube-api-access-knngm\" (UniqueName: \"kubernetes.io/projected/c56cc3d3-d13b-4b98-a94f-4de0d7d6f796-kube-api-access-knngm\") pod \"c56cc3d3-d13b-4b98-a94f-4de0d7d6f796\" (UID: \"c56cc3d3-d13b-4b98-a94f-4de0d7d6f796\") " Oct 31 00:50:54.131378 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.131348 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6131e5be-2a37-463e-ad2b-d1f60b831a7e-prometheus-client-cert" pod "6131e5be-2a37-463e-ad2b-d1f60b831a7e" (UID: "6131e5be-2a37-463e-ad2b-d1f60b831a7e"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:54.131930 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.131910 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6131e5be-2a37-463e-ad2b-d1f60b831a7e-prometheus-ca" pod "6131e5be-2a37-463e-ad2b-d1f60b831a7e" (UID: "6131e5be-2a37-463e-ad2b-d1f60b831a7e"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:54.132039 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.131906 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c56cc3d3-d13b-4b98-a94f-4de0d7d6f796-config" pod "c56cc3d3-d13b-4b98-a94f-4de0d7d6f796" (UID: "c56cc3d3-d13b-4b98-a94f-4de0d7d6f796"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:54.132921 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.132890 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6131e5be-2a37-463e-ad2b-d1f60b831a7e-tls" pod "6131e5be-2a37-463e-ad2b-d1f60b831a7e" (UID: "6131e5be-2a37-463e-ad2b-d1f60b831a7e"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:54.133034 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.132944 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c56cc3d3-d13b-4b98-a94f-4de0d7d6f796-kube-api-access-knngm" pod "c56cc3d3-d13b-4b98-a94f-4de0d7d6f796" (UID: "c56cc3d3-d13b-4b98-a94f-4de0d7d6f796"). InnerVolumeSpecName "kube-api-access-knngm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:54.136744 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.134833 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6131e5be-2a37-463e-ad2b-d1f60b831a7e-kube-api-access-7spdk" pod "6131e5be-2a37-463e-ad2b-d1f60b831a7e" (UID: "6131e5be-2a37-463e-ad2b-d1f60b831a7e"). InnerVolumeSpecName "kube-api-access-7spdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:54.137803 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.137095 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c56cc3d3-d13b-4b98-a94f-4de0d7d6f796-tls" pod "c56cc3d3-d13b-4b98-a94f-4de0d7d6f796" (UID: "c56cc3d3-d13b-4b98-a94f-4de0d7d6f796"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:54.137803 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.137157 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6131e5be-2a37-463e-ad2b-d1f60b831a7e-kube-rbac-proxy-config" pod "6131e5be-2a37-463e-ad2b-d1f60b831a7e" (UID: "6131e5be-2a37-463e-ad2b-d1f60b831a7e"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:54.229646 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.229564 5274 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6131e5be-2a37-463e-ad2b-d1f60b831a7e-kube-rbac-proxy-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.229646 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.229597 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knngm\" (UniqueName: \"kubernetes.io/projected/c56cc3d3-d13b-4b98-a94f-4de0d7d6f796-kube-api-access-knngm\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.229646 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.229614 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6131e5be-2a37-463e-ad2b-d1f60b831a7e-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.229646 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.229629 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c56cc3d3-d13b-4b98-a94f-4de0d7d6f796-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.229646 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.229644 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/6131e5be-2a37-463e-ad2b-d1f60b831a7e-prometheus-ca\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.229922 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.229658 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/6131e5be-2a37-463e-ad2b-d1f60b831a7e-prometheus-client-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.229922 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.229672 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7spdk\" (UniqueName: \"kubernetes.io/projected/6131e5be-2a37-463e-ad2b-d1f60b831a7e-kube-api-access-7spdk\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.229922 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.229687 5274 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c56cc3d3-d13b-4b98-a94f-4de0d7d6f796-tls\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.436766 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.436734 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:54.444259 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.444239 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:54.521969 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.521881 5274 generic.go:356] "Generic (PLEG): container finished" podID="6131e5be-2a37-463e-ad2b-d1f60b831a7e" containerID="e4631160363c736227355b173232b8346029c508918585168f349944c235a53f" exitCode=0 Oct 31 00:50:54.521969 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.521947 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-fmzfj" event={"ID":"6131e5be-2a37-463e-ad2b-d1f60b831a7e","Type":"ContainerDied","Data":"e4631160363c736227355b173232b8346029c508918585168f349944c235a53f"} Oct 31 00:50:54.523015 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.522030 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-fmzfj" event={"ID":"6131e5be-2a37-463e-ad2b-d1f60b831a7e","Type":"ContainerDied","Data":"26340255553df18cc716fa1dcd899090e9fe54b6270d56cd95a571c170f0f3ba"} Oct 31 00:50:54.523015 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.522034 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-fmzfj" Oct 31 00:50:54.523015 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.522082 5274 scope.go:122] "RemoveContainer" containerID="e4631160363c736227355b173232b8346029c508918585168f349944c235a53f" Oct 31 00:50:54.528055 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.528029 5274 generic.go:356] "Generic (PLEG): container finished" podID="6cea6fd7-14ad-45ce-b408-6042192c0a97" containerID="cd9d621ef5db0864be3004de60359a02ccfed1c46dd7b073fbb26e9a2df1f6e2" exitCode=0 Oct 31 00:50:54.528160 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.528104 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"6cea6fd7-14ad-45ce-b408-6042192c0a97","Type":"ContainerDied","Data":"cd9d621ef5db0864be3004de60359a02ccfed1c46dd7b073fbb26e9a2df1f6e2"} Oct 31 00:50:54.529661 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.529635 5274 generic.go:356] "Generic (PLEG): container finished" podID="c56cc3d3-d13b-4b98-a94f-4de0d7d6f796" containerID="15ad16eb9092c758f9e004a6963c0ac0938dd4dc623cee3695daf996810cba15" exitCode=0 Oct 31 00:50:54.529751 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.529710 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8gsx4" event={"ID":"c56cc3d3-d13b-4b98-a94f-4de0d7d6f796","Type":"ContainerDied","Data":"15ad16eb9092c758f9e004a6963c0ac0938dd4dc623cee3695daf996810cba15"} Oct 31 00:50:54.529801 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.529749 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8gsx4" event={"ID":"c56cc3d3-d13b-4b98-a94f-4de0d7d6f796","Type":"ContainerDied","Data":"89107d1e64c4ad045f31135f20a2bee7a7988e7a6c80b590a61600f80d5f144c"} Oct 31 00:50:54.529844 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.529835 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-8gsx4" Oct 31 00:50:54.531501 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.531479 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-secret-prometheus-web-tls-ca\") pod \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " Oct 31 00:50:54.531633 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.531535 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db1066ed-101f-436d-916e-88fea4540e87-kube-api-access-msgf5\" (UniqueName: \"kubernetes.io/projected/db1066ed-101f-436d-916e-88fea4540e87-kube-api-access-msgf5\") pod \"db1066ed-101f-436d-916e-88fea4540e87\" (UID: \"db1066ed-101f-436d-916e-88fea4540e87\") " Oct 31 00:50:54.531633 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.531569 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db1066ed-101f-436d-916e-88fea4540e87-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/db1066ed-101f-436d-916e-88fea4540e87-otlp-tls-certs\") pod \"db1066ed-101f-436d-916e-88fea4540e87\" (UID: \"db1066ed-101f-436d-916e-88fea4540e87\") " Oct 31 00:50:54.531633 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.531606 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-kube-api-access-mm592\" (UniqueName: \"kubernetes.io/projected/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-kube-api-access-mm592\") pod \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " Oct 31 00:50:54.531762 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.531633 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-web-config\" (UniqueName: \"kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-web-config\") pod \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " Oct 31 00:50:54.531762 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.531718 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") pod \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " Oct 31 00:50:54.531832 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.531756 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " Oct 31 00:50:54.531832 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.531796 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-thanos-prometheus-http-client-file\") pod \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " Oct 31 00:50:54.531901 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.531832 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " Oct 31 00:50:54.531901 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.531861 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-config\" (UniqueName: \"kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-config\") pod \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " Oct 31 00:50:54.531901 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.531893 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " Oct 31 00:50:54.532047 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.531973 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " Oct 31 00:50:54.532047 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.532013 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " Oct 31 00:50:54.532163 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.532055 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-tls-assets\" (UniqueName: \"kubernetes.io/projected/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-tls-assets\") pod \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " Oct 31 00:50:54.532163 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.532085 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-config-out\" (UniqueName: \"kubernetes.io/empty-dir/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-config-out\") pod \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\" (UID: \"094e5b88-2f67-4ec4-9847-f03dfd7eca5b\") " Oct 31 00:50:54.532163 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.532123 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db1066ed-101f-436d-916e-88fea4540e87-otc-internal\" (UniqueName: \"kubernetes.io/configmap/db1066ed-101f-436d-916e-88fea4540e87-otc-internal\") pod \"db1066ed-101f-436d-916e-88fea4540e87\" (UID: \"db1066ed-101f-436d-916e-88fea4540e87\") " Oct 31 00:50:54.532330 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.532174 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db1066ed-101f-436d-916e-88fea4540e87-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/db1066ed-101f-436d-916e-88fea4540e87-monitoring-tls-certs\") pod \"db1066ed-101f-436d-916e-88fea4540e87\" (UID: \"db1066ed-101f-436d-916e-88fea4540e87\") " Oct 31 00:50:54.532330 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.532228 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db1066ed-101f-436d-916e-88fea4540e87-tls-certs\" (UniqueName: \"kubernetes.io/secret/db1066ed-101f-436d-916e-88fea4540e87-tls-certs\") pod \"db1066ed-101f-436d-916e-88fea4540e87\" (UID: \"db1066ed-101f-436d-916e-88fea4540e87\") " Oct 31 00:50:54.532566 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.532491 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db1066ed-101f-436d-916e-88fea4540e87-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/db1066ed-101f-436d-916e-88fea4540e87-data-science-collector-ta-client-cert\") pod \"db1066ed-101f-436d-916e-88fea4540e87\" (UID: \"db1066ed-101f-436d-916e-88fea4540e87\") " Oct 31 00:50:54.533191 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.533144 5274 generic.go:356] "Generic (PLEG): container finished" podID="db1066ed-101f-436d-916e-88fea4540e87" containerID="26e8386dc53d861ae4a79be0d6da7cca684f6e4a95dbbac3b63ba4472cab1109" exitCode=0 Oct 31 00:50:54.533191 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.533182 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-prometheus-data-science-monitoringstack-rulefiles-1" pod "094e5b88-2f67-4ec4-9847-f03dfd7eca5b" (UID: "094e5b88-2f67-4ec4-9847-f03dfd7eca5b"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:54.533346 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.533304 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"db1066ed-101f-436d-916e-88fea4540e87","Type":"ContainerDied","Data":"26e8386dc53d861ae4a79be0d6da7cca684f6e4a95dbbac3b63ba4472cab1109"} Oct 31 00:50:54.533407 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.533349 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"db1066ed-101f-436d-916e-88fea4540e87","Type":"ContainerDied","Data":"9eb6222cccc10ba23a4537fe5732f252a9555ae874a32e128b3f893b26304237"} Oct 31 00:50:54.533407 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.533366 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:50:54.536251 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.535011 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db1066ed-101f-436d-916e-88fea4540e87-otc-internal" pod "db1066ed-101f-436d-916e-88fea4540e87" (UID: "db1066ed-101f-436d-916e-88fea4540e87"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:54.537364 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.537332 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "094e5b88-2f67-4ec4-9847-f03dfd7eca5b" (UID: "094e5b88-2f67-4ec4-9847-f03dfd7eca5b"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:54.538157 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.537669 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-prometheus-data-science-monitoringstack-rulefiles-2" pod "094e5b88-2f67-4ec4-9847-f03dfd7eca5b" (UID: "094e5b88-2f67-4ec4-9847-f03dfd7eca5b"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:54.539425 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.538760 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db1066ed-101f-436d-916e-88fea4540e87-monitoring-tls-certs" pod "db1066ed-101f-436d-916e-88fea4540e87" (UID: "db1066ed-101f-436d-916e-88fea4540e87"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:54.541109 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.541084 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db1066ed-101f-436d-916e-88fea4540e87-tls-certs" pod "db1066ed-101f-436d-916e-88fea4540e87" (UID: "db1066ed-101f-436d-916e-88fea4540e87"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:54.541109 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.541091 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "094e5b88-2f67-4ec4-9847-f03dfd7eca5b" (UID: "094e5b88-2f67-4ec4-9847-f03dfd7eca5b"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:54.541373 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.541344 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-prometheus-data-science-monitoringstack-rulefiles-0" pod "094e5b88-2f67-4ec4-9847-f03dfd7eca5b" (UID: "094e5b88-2f67-4ec4-9847-f03dfd7eca5b"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:54.546551 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.546221 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-tls-assets" pod "094e5b88-2f67-4ec4-9847-f03dfd7eca5b" (UID: "094e5b88-2f67-4ec4-9847-f03dfd7eca5b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:54.546551 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.546300 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db1066ed-101f-436d-916e-88fea4540e87-otlp-tls-certs" pod "db1066ed-101f-436d-916e-88fea4540e87" (UID: "db1066ed-101f-436d-916e-88fea4540e87"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:54.546929 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.546903 5274 generic.go:356] "Generic (PLEG): container finished" podID="094e5b88-2f67-4ec4-9847-f03dfd7eca5b" containerID="e6eb5563f443744d815dd7b6126708784884f4ff4e89d043adf39cc487342adc" exitCode=0 Oct 31 00:50:54.547071 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.546933 5274 generic.go:356] "Generic (PLEG): container finished" podID="094e5b88-2f67-4ec4-9847-f03dfd7eca5b" containerID="6f46ceff18145e6db5f1dc29192fadcd43c3bd795aba262dac20a290d1d669e6" exitCode=0 Oct 31 00:50:54.547071 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.546944 5274 generic.go:356] "Generic (PLEG): container finished" podID="094e5b88-2f67-4ec4-9847-f03dfd7eca5b" containerID="ad42ac969a0002b6039492ed99ef0f6913c3a58ee1153a720d98d32ae4b58ff5" exitCode=0 Oct 31 00:50:54.547071 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.547008 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"094e5b88-2f67-4ec4-9847-f03dfd7eca5b","Type":"ContainerDied","Data":"e6eb5563f443744d815dd7b6126708784884f4ff4e89d043adf39cc487342adc"} Oct 31 00:50:54.547071 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.547066 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"094e5b88-2f67-4ec4-9847-f03dfd7eca5b","Type":"ContainerDied","Data":"6f46ceff18145e6db5f1dc29192fadcd43c3bd795aba262dac20a290d1d669e6"} Oct 31 00:50:54.547296 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.547106 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"094e5b88-2f67-4ec4-9847-f03dfd7eca5b","Type":"ContainerDied","Data":"ad42ac969a0002b6039492ed99ef0f6913c3a58ee1153a720d98d32ae4b58ff5"} Oct 31 00:50:54.547296 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.547143 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"094e5b88-2f67-4ec4-9847-f03dfd7eca5b","Type":"ContainerDied","Data":"56534a8d5c663fff8a6e75f374a07d555ce7650e5e9579fe793c9dd101844fc3"} Oct 31 00:50:54.547296 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.547251 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 31 00:50:54.547482 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.547458 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-config" pod "094e5b88-2f67-4ec4-9847-f03dfd7eca5b" (UID: "094e5b88-2f67-4ec4-9847-f03dfd7eca5b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:54.547890 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.547855 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db1066ed-101f-436d-916e-88fea4540e87-data-science-collector-ta-client-cert" pod "db1066ed-101f-436d-916e-88fea4540e87" (UID: "db1066ed-101f-436d-916e-88fea4540e87"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:54.548041 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.547983 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db1066ed-101f-436d-916e-88fea4540e87-kube-api-access-msgf5" pod "db1066ed-101f-436d-916e-88fea4540e87" (UID: "db1066ed-101f-436d-916e-88fea4540e87"). InnerVolumeSpecName "kube-api-access-msgf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:54.548041 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.547341 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-kube-api-access-mm592" pod "094e5b88-2f67-4ec4-9847-f03dfd7eca5b" (UID: "094e5b88-2f67-4ec4-9847-f03dfd7eca5b"). InnerVolumeSpecName "kube-api-access-mm592". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:54.548176 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.548103 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a" pod "094e5b88-2f67-4ec4-9847-f03dfd7eca5b" (UID: "094e5b88-2f67-4ec4-9847-f03dfd7eca5b"). InnerVolumeSpecName "pvc-63e34709-8d85-4142-b53a-f655698606ff". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:50:54.551658 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.551570 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-secret-prometheus-web-tls-ca" pod "094e5b88-2f67-4ec4-9847-f03dfd7eca5b" (UID: "094e5b88-2f67-4ec4-9847-f03dfd7eca5b"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:54.551658 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.551465 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-thanos-prometheus-http-client-file" pod "094e5b88-2f67-4ec4-9847-f03dfd7eca5b" (UID: "094e5b88-2f67-4ec4-9847-f03dfd7eca5b"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:54.552282 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.552259 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-config-out" pod "094e5b88-2f67-4ec4-9847-f03dfd7eca5b" (UID: "094e5b88-2f67-4ec4-9847-f03dfd7eca5b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:54.553746 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.553714 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-fmzfj"] Oct 31 00:50:54.559701 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.559680 5274 scope.go:122] "RemoveContainer" containerID="e4631160363c736227355b173232b8346029c508918585168f349944c235a53f" Oct 31 00:50:54.560108 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:50:54.560081 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4631160363c736227355b173232b8346029c508918585168f349944c235a53f\": container with ID starting with e4631160363c736227355b173232b8346029c508918585168f349944c235a53f not found: ID does not exist" containerID="e4631160363c736227355b173232b8346029c508918585168f349944c235a53f" Oct 31 00:50:54.560195 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.560117 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4631160363c736227355b173232b8346029c508918585168f349944c235a53f"} err="failed to get container status \"e4631160363c736227355b173232b8346029c508918585168f349944c235a53f\": rpc error: code = NotFound desc = could not find container \"e4631160363c736227355b173232b8346029c508918585168f349944c235a53f\": container with ID starting with e4631160363c736227355b173232b8346029c508918585168f349944c235a53f not found: ID does not exist" Oct 31 00:50:54.560195 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.560137 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-fmzfj"] Oct 31 00:50:54.560195 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.560142 5274 scope.go:122] "RemoveContainer" containerID="15ad16eb9092c758f9e004a6963c0ac0938dd4dc623cee3695daf996810cba15" Oct 31 00:50:54.567414 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.567382 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-web-config" pod "094e5b88-2f67-4ec4-9847-f03dfd7eca5b" (UID: "094e5b88-2f67-4ec4-9847-f03dfd7eca5b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:54.570196 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.570182 5274 scope.go:122] "RemoveContainer" containerID="15ad16eb9092c758f9e004a6963c0ac0938dd4dc623cee3695daf996810cba15" Oct 31 00:50:54.570494 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:50:54.570476 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15ad16eb9092c758f9e004a6963c0ac0938dd4dc623cee3695daf996810cba15\": container with ID starting with 15ad16eb9092c758f9e004a6963c0ac0938dd4dc623cee3695daf996810cba15 not found: ID does not exist" containerID="15ad16eb9092c758f9e004a6963c0ac0938dd4dc623cee3695daf996810cba15" Oct 31 00:50:54.570590 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.570497 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15ad16eb9092c758f9e004a6963c0ac0938dd4dc623cee3695daf996810cba15"} err="failed to get container status \"15ad16eb9092c758f9e004a6963c0ac0938dd4dc623cee3695daf996810cba15\": rpc error: code = NotFound desc = could not find container \"15ad16eb9092c758f9e004a6963c0ac0938dd4dc623cee3695daf996810cba15\": container with ID starting with 15ad16eb9092c758f9e004a6963c0ac0938dd4dc623cee3695daf996810cba15 not found: ID does not exist" Oct 31 00:50:54.570590 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.570513 5274 scope.go:122] "RemoveContainer" containerID="26e8386dc53d861ae4a79be0d6da7cca684f6e4a95dbbac3b63ba4472cab1109" Oct 31 00:50:54.571746 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.571707 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-8gsx4"] Oct 31 00:50:54.576635 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.576615 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-8gsx4"] Oct 31 00:50:54.579908 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.579884 5274 scope.go:122] "RemoveContainer" containerID="26e8386dc53d861ae4a79be0d6da7cca684f6e4a95dbbac3b63ba4472cab1109" Oct 31 00:50:54.580205 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:50:54.580178 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26e8386dc53d861ae4a79be0d6da7cca684f6e4a95dbbac3b63ba4472cab1109\": container with ID starting with 26e8386dc53d861ae4a79be0d6da7cca684f6e4a95dbbac3b63ba4472cab1109 not found: ID does not exist" containerID="26e8386dc53d861ae4a79be0d6da7cca684f6e4a95dbbac3b63ba4472cab1109" Oct 31 00:50:54.580268 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.580214 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26e8386dc53d861ae4a79be0d6da7cca684f6e4a95dbbac3b63ba4472cab1109"} err="failed to get container status \"26e8386dc53d861ae4a79be0d6da7cca684f6e4a95dbbac3b63ba4472cab1109\": rpc error: code = NotFound desc = could not find container \"26e8386dc53d861ae4a79be0d6da7cca684f6e4a95dbbac3b63ba4472cab1109\": container with ID starting with 26e8386dc53d861ae4a79be0d6da7cca684f6e4a95dbbac3b63ba4472cab1109 not found: ID does not exist" Oct 31 00:50:54.580268 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.580235 5274 scope.go:122] "RemoveContainer" containerID="e6eb5563f443744d815dd7b6126708784884f4ff4e89d043adf39cc487342adc" Oct 31 00:50:54.587758 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.587742 5274 scope.go:122] "RemoveContainer" containerID="6f46ceff18145e6db5f1dc29192fadcd43c3bd795aba262dac20a290d1d669e6" Oct 31 00:50:54.598228 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.598209 5274 scope.go:122] "RemoveContainer" containerID="ad42ac969a0002b6039492ed99ef0f6913c3a58ee1153a720d98d32ae4b58ff5" Oct 31 00:50:54.606319 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.606297 5274 scope.go:122] "RemoveContainer" containerID="d2c7e3e8613de4cdc5fcc833b8ec145e800ae8a809e5506df0ad5077f9477e3e" Oct 31 00:50:54.614560 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.614541 5274 scope.go:122] "RemoveContainer" containerID="e6eb5563f443744d815dd7b6126708784884f4ff4e89d043adf39cc487342adc" Oct 31 00:50:54.614878 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:50:54.614850 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6eb5563f443744d815dd7b6126708784884f4ff4e89d043adf39cc487342adc\": container with ID starting with e6eb5563f443744d815dd7b6126708784884f4ff4e89d043adf39cc487342adc not found: ID does not exist" containerID="e6eb5563f443744d815dd7b6126708784884f4ff4e89d043adf39cc487342adc" Oct 31 00:50:54.614942 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.614885 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6eb5563f443744d815dd7b6126708784884f4ff4e89d043adf39cc487342adc"} err="failed to get container status \"e6eb5563f443744d815dd7b6126708784884f4ff4e89d043adf39cc487342adc\": rpc error: code = NotFound desc = could not find container \"e6eb5563f443744d815dd7b6126708784884f4ff4e89d043adf39cc487342adc\": container with ID starting with e6eb5563f443744d815dd7b6126708784884f4ff4e89d043adf39cc487342adc not found: ID does not exist" Oct 31 00:50:54.614942 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.614904 5274 scope.go:122] "RemoveContainer" containerID="6f46ceff18145e6db5f1dc29192fadcd43c3bd795aba262dac20a290d1d669e6" Oct 31 00:50:54.615217 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:50:54.615198 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f46ceff18145e6db5f1dc29192fadcd43c3bd795aba262dac20a290d1d669e6\": container with ID starting with 6f46ceff18145e6db5f1dc29192fadcd43c3bd795aba262dac20a290d1d669e6 not found: ID does not exist" containerID="6f46ceff18145e6db5f1dc29192fadcd43c3bd795aba262dac20a290d1d669e6" Oct 31 00:50:54.615275 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.615225 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f46ceff18145e6db5f1dc29192fadcd43c3bd795aba262dac20a290d1d669e6"} err="failed to get container status \"6f46ceff18145e6db5f1dc29192fadcd43c3bd795aba262dac20a290d1d669e6\": rpc error: code = NotFound desc = could not find container \"6f46ceff18145e6db5f1dc29192fadcd43c3bd795aba262dac20a290d1d669e6\": container with ID starting with 6f46ceff18145e6db5f1dc29192fadcd43c3bd795aba262dac20a290d1d669e6 not found: ID does not exist" Oct 31 00:50:54.615275 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.615242 5274 scope.go:122] "RemoveContainer" containerID="ad42ac969a0002b6039492ed99ef0f6913c3a58ee1153a720d98d32ae4b58ff5" Oct 31 00:50:54.615526 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:50:54.615511 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad42ac969a0002b6039492ed99ef0f6913c3a58ee1153a720d98d32ae4b58ff5\": container with ID starting with ad42ac969a0002b6039492ed99ef0f6913c3a58ee1153a720d98d32ae4b58ff5 not found: ID does not exist" containerID="ad42ac969a0002b6039492ed99ef0f6913c3a58ee1153a720d98d32ae4b58ff5" Oct 31 00:50:54.615569 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.615530 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad42ac969a0002b6039492ed99ef0f6913c3a58ee1153a720d98d32ae4b58ff5"} err="failed to get container status \"ad42ac969a0002b6039492ed99ef0f6913c3a58ee1153a720d98d32ae4b58ff5\": rpc error: code = NotFound desc = could not find container \"ad42ac969a0002b6039492ed99ef0f6913c3a58ee1153a720d98d32ae4b58ff5\": container with ID starting with ad42ac969a0002b6039492ed99ef0f6913c3a58ee1153a720d98d32ae4b58ff5 not found: ID does not exist" Oct 31 00:50:54.615569 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.615543 5274 scope.go:122] "RemoveContainer" containerID="d2c7e3e8613de4cdc5fcc833b8ec145e800ae8a809e5506df0ad5077f9477e3e" Oct 31 00:50:54.615783 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:50:54.615767 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2c7e3e8613de4cdc5fcc833b8ec145e800ae8a809e5506df0ad5077f9477e3e\": container with ID starting with d2c7e3e8613de4cdc5fcc833b8ec145e800ae8a809e5506df0ad5077f9477e3e not found: ID does not exist" containerID="d2c7e3e8613de4cdc5fcc833b8ec145e800ae8a809e5506df0ad5077f9477e3e" Oct 31 00:50:54.615829 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.615787 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2c7e3e8613de4cdc5fcc833b8ec145e800ae8a809e5506df0ad5077f9477e3e"} err="failed to get container status \"d2c7e3e8613de4cdc5fcc833b8ec145e800ae8a809e5506df0ad5077f9477e3e\": rpc error: code = NotFound desc = could not find container \"d2c7e3e8613de4cdc5fcc833b8ec145e800ae8a809e5506df0ad5077f9477e3e\": container with ID starting with d2c7e3e8613de4cdc5fcc833b8ec145e800ae8a809e5506df0ad5077f9477e3e not found: ID does not exist" Oct 31 00:50:54.615829 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.615800 5274 scope.go:122] "RemoveContainer" containerID="e6eb5563f443744d815dd7b6126708784884f4ff4e89d043adf39cc487342adc" Oct 31 00:50:54.616052 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.616024 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6eb5563f443744d815dd7b6126708784884f4ff4e89d043adf39cc487342adc"} err="failed to get container status \"e6eb5563f443744d815dd7b6126708784884f4ff4e89d043adf39cc487342adc\": rpc error: code = NotFound desc = could not find container \"e6eb5563f443744d815dd7b6126708784884f4ff4e89d043adf39cc487342adc\": container with ID starting with e6eb5563f443744d815dd7b6126708784884f4ff4e89d043adf39cc487342adc not found: ID does not exist" Oct 31 00:50:54.616052 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.616052 5274 scope.go:122] "RemoveContainer" containerID="6f46ceff18145e6db5f1dc29192fadcd43c3bd795aba262dac20a290d1d669e6" Oct 31 00:50:54.616323 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.616299 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f46ceff18145e6db5f1dc29192fadcd43c3bd795aba262dac20a290d1d669e6"} err="failed to get container status \"6f46ceff18145e6db5f1dc29192fadcd43c3bd795aba262dac20a290d1d669e6\": rpc error: code = NotFound desc = could not find container \"6f46ceff18145e6db5f1dc29192fadcd43c3bd795aba262dac20a290d1d669e6\": container with ID starting with 6f46ceff18145e6db5f1dc29192fadcd43c3bd795aba262dac20a290d1d669e6 not found: ID does not exist" Oct 31 00:50:54.616403 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.616325 5274 scope.go:122] "RemoveContainer" containerID="ad42ac969a0002b6039492ed99ef0f6913c3a58ee1153a720d98d32ae4b58ff5" Oct 31 00:50:54.616588 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.616569 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad42ac969a0002b6039492ed99ef0f6913c3a58ee1153a720d98d32ae4b58ff5"} err="failed to get container status \"ad42ac969a0002b6039492ed99ef0f6913c3a58ee1153a720d98d32ae4b58ff5\": rpc error: code = NotFound desc = could not find container \"ad42ac969a0002b6039492ed99ef0f6913c3a58ee1153a720d98d32ae4b58ff5\": container with ID starting with ad42ac969a0002b6039492ed99ef0f6913c3a58ee1153a720d98d32ae4b58ff5 not found: ID does not exist" Oct 31 00:50:54.616629 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.616599 5274 scope.go:122] "RemoveContainer" containerID="d2c7e3e8613de4cdc5fcc833b8ec145e800ae8a809e5506df0ad5077f9477e3e" Oct 31 00:50:54.616866 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.616847 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2c7e3e8613de4cdc5fcc833b8ec145e800ae8a809e5506df0ad5077f9477e3e"} err="failed to get container status \"d2c7e3e8613de4cdc5fcc833b8ec145e800ae8a809e5506df0ad5077f9477e3e\": rpc error: code = NotFound desc = could not find container \"d2c7e3e8613de4cdc5fcc833b8ec145e800ae8a809e5506df0ad5077f9477e3e\": container with ID starting with d2c7e3e8613de4cdc5fcc833b8ec145e800ae8a809e5506df0ad5077f9477e3e not found: ID does not exist" Oct 31 00:50:54.616942 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.616866 5274 scope.go:122] "RemoveContainer" containerID="e6eb5563f443744d815dd7b6126708784884f4ff4e89d043adf39cc487342adc" Oct 31 00:50:54.617176 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.617158 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6eb5563f443744d815dd7b6126708784884f4ff4e89d043adf39cc487342adc"} err="failed to get container status \"e6eb5563f443744d815dd7b6126708784884f4ff4e89d043adf39cc487342adc\": rpc error: code = NotFound desc = could not find container \"e6eb5563f443744d815dd7b6126708784884f4ff4e89d043adf39cc487342adc\": container with ID starting with e6eb5563f443744d815dd7b6126708784884f4ff4e89d043adf39cc487342adc not found: ID does not exist" Oct 31 00:50:54.617246 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.617177 5274 scope.go:122] "RemoveContainer" containerID="6f46ceff18145e6db5f1dc29192fadcd43c3bd795aba262dac20a290d1d669e6" Oct 31 00:50:54.617488 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.617434 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f46ceff18145e6db5f1dc29192fadcd43c3bd795aba262dac20a290d1d669e6"} err="failed to get container status \"6f46ceff18145e6db5f1dc29192fadcd43c3bd795aba262dac20a290d1d669e6\": rpc error: code = NotFound desc = could not find container \"6f46ceff18145e6db5f1dc29192fadcd43c3bd795aba262dac20a290d1d669e6\": container with ID starting with 6f46ceff18145e6db5f1dc29192fadcd43c3bd795aba262dac20a290d1d669e6 not found: ID does not exist" Oct 31 00:50:54.617578 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.617491 5274 scope.go:122] "RemoveContainer" containerID="ad42ac969a0002b6039492ed99ef0f6913c3a58ee1153a720d98d32ae4b58ff5" Oct 31 00:50:54.617751 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.617733 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad42ac969a0002b6039492ed99ef0f6913c3a58ee1153a720d98d32ae4b58ff5"} err="failed to get container status \"ad42ac969a0002b6039492ed99ef0f6913c3a58ee1153a720d98d32ae4b58ff5\": rpc error: code = NotFound desc = could not find container \"ad42ac969a0002b6039492ed99ef0f6913c3a58ee1153a720d98d32ae4b58ff5\": container with ID starting with ad42ac969a0002b6039492ed99ef0f6913c3a58ee1153a720d98d32ae4b58ff5 not found: ID does not exist" Oct 31 00:50:54.617821 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.617751 5274 scope.go:122] "RemoveContainer" containerID="d2c7e3e8613de4cdc5fcc833b8ec145e800ae8a809e5506df0ad5077f9477e3e" Oct 31 00:50:54.618109 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.618031 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2c7e3e8613de4cdc5fcc833b8ec145e800ae8a809e5506df0ad5077f9477e3e"} err="failed to get container status \"d2c7e3e8613de4cdc5fcc833b8ec145e800ae8a809e5506df0ad5077f9477e3e\": rpc error: code = NotFound desc = could not find container \"d2c7e3e8613de4cdc5fcc833b8ec145e800ae8a809e5506df0ad5077f9477e3e\": container with ID starting with d2c7e3e8613de4cdc5fcc833b8ec145e800ae8a809e5506df0ad5077f9477e3e not found: ID does not exist" Oct 31 00:50:54.633688 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.633664 5274 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/db1066ed-101f-436d-916e-88fea4540e87-monitoring-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.633688 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.633686 5274 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db1066ed-101f-436d-916e-88fea4540e87-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.633858 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.633700 5274 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/db1066ed-101f-436d-916e-88fea4540e87-data-science-collector-ta-client-cert\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.633858 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.633713 5274 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-secret-prometheus-web-tls-ca\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.633858 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.633722 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msgf5\" (UniqueName: \"kubernetes.io/projected/db1066ed-101f-436d-916e-88fea4540e87-kube-api-access-msgf5\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.633858 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.633731 5274 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/db1066ed-101f-436d-916e-88fea4540e87-otlp-tls-certs\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.633858 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.633739 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm592\" (UniqueName: \"kubernetes.io/projected/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-kube-api-access-mm592\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.633858 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.633747 5274 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-web-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.633858 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.633775 5274 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" " Oct 31 00:50:54.633858 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.633790 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.633858 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.633800 5274 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.633858 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.633815 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.633858 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.633829 5274 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.633858 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.633844 5274 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.633858 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.633856 5274 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.634420 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.633872 5274 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.634420 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.633885 5274 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-tls-assets\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.634420 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.633899 5274 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/094e5b88-2f67-4ec4-9847-f03dfd7eca5b-config-out\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.634420 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.633907 5274 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/db1066ed-101f-436d-916e-88fea4540e87-otc-internal\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.648186 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.648171 5274 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-63e34709-8d85-4142-b53a-f655698606ff" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a") on node "ip-10-0-129-70.us-east-2.compute.internal" Oct 31 00:50:54.687939 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.687923 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:50:54.734787 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.734762 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cea6fd7-14ad-45ce-b408-6042192c0a97-web-config\" (UniqueName: \"kubernetes.io/secret/6cea6fd7-14ad-45ce-b408-6042192c0a97-web-config\") pod \"6cea6fd7-14ad-45ce-b408-6042192c0a97\" (UID: \"6cea6fd7-14ad-45ce-b408-6042192c0a97\") " Oct 31 00:50:54.735176 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.734813 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cea6fd7-14ad-45ce-b408-6042192c0a97-tls-assets\" (UniqueName: \"kubernetes.io/projected/6cea6fd7-14ad-45ce-b408-6042192c0a97-tls-assets\") pod \"6cea6fd7-14ad-45ce-b408-6042192c0a97\" (UID: \"6cea6fd7-14ad-45ce-b408-6042192c0a97\") " Oct 31 00:50:54.735176 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.734869 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cea6fd7-14ad-45ce-b408-6042192c0a97-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6cea6fd7-14ad-45ce-b408-6042192c0a97-cluster-tls-config\") pod \"6cea6fd7-14ad-45ce-b408-6042192c0a97\" (UID: \"6cea6fd7-14ad-45ce-b408-6042192c0a97\") " Oct 31 00:50:54.735176 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.734973 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cea6fd7-14ad-45ce-b408-6042192c0a97-kube-api-access-8wbr6\" (UniqueName: \"kubernetes.io/projected/6cea6fd7-14ad-45ce-b408-6042192c0a97-kube-api-access-8wbr6\") pod \"6cea6fd7-14ad-45ce-b408-6042192c0a97\" (UID: \"6cea6fd7-14ad-45ce-b408-6042192c0a97\") " Oct 31 00:50:54.735176 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.735019 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cea6fd7-14ad-45ce-b408-6042192c0a97-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/6cea6fd7-14ad-45ce-b408-6042192c0a97-alertmanager-data-science-monitoringstack-db\") pod \"6cea6fd7-14ad-45ce-b408-6042192c0a97\" (UID: \"6cea6fd7-14ad-45ce-b408-6042192c0a97\") " Oct 31 00:50:54.735176 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.735054 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cea6fd7-14ad-45ce-b408-6042192c0a97-config-volume\" (UniqueName: \"kubernetes.io/secret/6cea6fd7-14ad-45ce-b408-6042192c0a97-config-volume\") pod \"6cea6fd7-14ad-45ce-b408-6042192c0a97\" (UID: \"6cea6fd7-14ad-45ce-b408-6042192c0a97\") " Oct 31 00:50:54.735176 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.735101 5274 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cea6fd7-14ad-45ce-b408-6042192c0a97-config-out\" (UniqueName: \"kubernetes.io/empty-dir/6cea6fd7-14ad-45ce-b408-6042192c0a97-config-out\") pod \"6cea6fd7-14ad-45ce-b408-6042192c0a97\" (UID: \"6cea6fd7-14ad-45ce-b408-6042192c0a97\") " Oct 31 00:50:54.735538 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.735334 5274 reconciler_common.go:299] "Volume detached for volume \"pvc-63e34709-8d85-4142-b53a-f655698606ff\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0898f33c7e5b7835a\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"csi-3beb91b3ebf177571ca474589da3560cd18787cfa50b2a0b9bfef85661e6cf21\"" Oct 31 00:50:54.735538 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.735345 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cea6fd7-14ad-45ce-b408-6042192c0a97-alertmanager-data-science-monitoringstack-db" pod "6cea6fd7-14ad-45ce-b408-6042192c0a97" (UID: "6cea6fd7-14ad-45ce-b408-6042192c0a97"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:54.737210 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.737183 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cea6fd7-14ad-45ce-b408-6042192c0a97-config-volume" pod "6cea6fd7-14ad-45ce-b408-6042192c0a97" (UID: "6cea6fd7-14ad-45ce-b408-6042192c0a97"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:54.737310 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.737297 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cea6fd7-14ad-45ce-b408-6042192c0a97-tls-assets" pod "6cea6fd7-14ad-45ce-b408-6042192c0a97" (UID: "6cea6fd7-14ad-45ce-b408-6042192c0a97"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:54.737458 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.737431 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cea6fd7-14ad-45ce-b408-6042192c0a97-kube-api-access-8wbr6" pod "6cea6fd7-14ad-45ce-b408-6042192c0a97" (UID: "6cea6fd7-14ad-45ce-b408-6042192c0a97"). InnerVolumeSpecName "kube-api-access-8wbr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:54.737539 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.737491 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cea6fd7-14ad-45ce-b408-6042192c0a97-config-out" pod "6cea6fd7-14ad-45ce-b408-6042192c0a97" (UID: "6cea6fd7-14ad-45ce-b408-6042192c0a97"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:54.742664 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.742644 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cea6fd7-14ad-45ce-b408-6042192c0a97-cluster-tls-config" pod "6cea6fd7-14ad-45ce-b408-6042192c0a97" (UID: "6cea6fd7-14ad-45ce-b408-6042192c0a97"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:54.749872 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.749849 5274 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cea6fd7-14ad-45ce-b408-6042192c0a97-web-config" pod "6cea6fd7-14ad-45ce-b408-6042192c0a97" (UID: "6cea6fd7-14ad-45ce-b408-6042192c0a97"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:54.835725 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.835701 5274 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wbr6\" (UniqueName: \"kubernetes.io/projected/6cea6fd7-14ad-45ce-b408-6042192c0a97-kube-api-access-8wbr6\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.835725 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.835724 5274 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/6cea6fd7-14ad-45ce-b408-6042192c0a97-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.835865 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.835735 5274 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6cea6fd7-14ad-45ce-b408-6042192c0a97-config-volume\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.835865 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.835745 5274 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6cea6fd7-14ad-45ce-b408-6042192c0a97-config-out\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.835865 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.835754 5274 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6cea6fd7-14ad-45ce-b408-6042192c0a97-web-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.835865 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.835762 5274 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6cea6fd7-14ad-45ce-b408-6042192c0a97-tls-assets\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.835865 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.835771 5274 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6cea6fd7-14ad-45ce-b408-6042192c0a97-cluster-tls-config\") on node \"ip-10-0-129-70.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.859394 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.859362 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:50:54.866699 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.866673 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:50:54.881115 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.881094 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:50:54.888764 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:54.888744 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 31 00:50:55.481706 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:55.481672 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="094e5b88-2f67-4ec4-9847-f03dfd7eca5b" path="/var/lib/kubelet/pods/094e5b88-2f67-4ec4-9847-f03dfd7eca5b/volumes" Oct 31 00:50:55.482220 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:55.482205 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6131e5be-2a37-463e-ad2b-d1f60b831a7e" path="/var/lib/kubelet/pods/6131e5be-2a37-463e-ad2b-d1f60b831a7e/volumes" Oct 31 00:50:55.482814 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:55.482799 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c56cc3d3-d13b-4b98-a94f-4de0d7d6f796" path="/var/lib/kubelet/pods/c56cc3d3-d13b-4b98-a94f-4de0d7d6f796/volumes" Oct 31 00:50:55.483183 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:55.483171 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db1066ed-101f-436d-916e-88fea4540e87" path="/var/lib/kubelet/pods/db1066ed-101f-436d-916e-88fea4540e87/volumes" Oct 31 00:50:55.553656 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:55.553634 5274 generic.go:356] "Generic (PLEG): container finished" podID="6cea6fd7-14ad-45ce-b408-6042192c0a97" containerID="70368cbfb832babccdf637b2863839a4daad2e3fae1a73052f16c011c3482629" exitCode=0 Oct 31 00:50:55.553785 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:55.553690 5274 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:50:55.553785 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:55.553720 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"6cea6fd7-14ad-45ce-b408-6042192c0a97","Type":"ContainerDied","Data":"70368cbfb832babccdf637b2863839a4daad2e3fae1a73052f16c011c3482629"} Oct 31 00:50:55.553877 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:55.553783 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"6cea6fd7-14ad-45ce-b408-6042192c0a97","Type":"ContainerDied","Data":"c189e0f3ce893a40d783015255c174218c89e282dae752fd9e9866489a11c82a"} Oct 31 00:50:55.553877 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:55.553823 5274 scope.go:122] "RemoveContainer" containerID="cd9d621ef5db0864be3004de60359a02ccfed1c46dd7b073fbb26e9a2df1f6e2" Oct 31 00:50:55.562883 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:55.562863 5274 scope.go:122] "RemoveContainer" containerID="70368cbfb832babccdf637b2863839a4daad2e3fae1a73052f16c011c3482629" Oct 31 00:50:55.570278 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:55.570265 5274 scope.go:122] "RemoveContainer" containerID="0b0c8265438d78f5dc0cd70a79231a3c92da53c6b6a03b3199811126e448a055" Oct 31 00:50:55.574156 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:55.574135 5274 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:50:55.577891 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:55.577874 5274 scope.go:122] "RemoveContainer" containerID="cd9d621ef5db0864be3004de60359a02ccfed1c46dd7b073fbb26e9a2df1f6e2" Oct 31 00:50:55.578167 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:50:55.578148 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd9d621ef5db0864be3004de60359a02ccfed1c46dd7b073fbb26e9a2df1f6e2\": container with ID starting with cd9d621ef5db0864be3004de60359a02ccfed1c46dd7b073fbb26e9a2df1f6e2 not found: ID does not exist" containerID="cd9d621ef5db0864be3004de60359a02ccfed1c46dd7b073fbb26e9a2df1f6e2" Oct 31 00:50:55.578248 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:55.578175 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd9d621ef5db0864be3004de60359a02ccfed1c46dd7b073fbb26e9a2df1f6e2"} err="failed to get container status \"cd9d621ef5db0864be3004de60359a02ccfed1c46dd7b073fbb26e9a2df1f6e2\": rpc error: code = NotFound desc = could not find container \"cd9d621ef5db0864be3004de60359a02ccfed1c46dd7b073fbb26e9a2df1f6e2\": container with ID starting with cd9d621ef5db0864be3004de60359a02ccfed1c46dd7b073fbb26e9a2df1f6e2 not found: ID does not exist" Oct 31 00:50:55.578248 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:55.578200 5274 scope.go:122] "RemoveContainer" containerID="70368cbfb832babccdf637b2863839a4daad2e3fae1a73052f16c011c3482629" Oct 31 00:50:55.578454 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:50:55.578435 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70368cbfb832babccdf637b2863839a4daad2e3fae1a73052f16c011c3482629\": container with ID starting with 70368cbfb832babccdf637b2863839a4daad2e3fae1a73052f16c011c3482629 not found: ID does not exist" containerID="70368cbfb832babccdf637b2863839a4daad2e3fae1a73052f16c011c3482629" Oct 31 00:50:55.578491 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:55.578460 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70368cbfb832babccdf637b2863839a4daad2e3fae1a73052f16c011c3482629"} err="failed to get container status \"70368cbfb832babccdf637b2863839a4daad2e3fae1a73052f16c011c3482629\": rpc error: code = NotFound desc = could not find container \"70368cbfb832babccdf637b2863839a4daad2e3fae1a73052f16c011c3482629\": container with ID starting with 70368cbfb832babccdf637b2863839a4daad2e3fae1a73052f16c011c3482629 not found: ID does not exist" Oct 31 00:50:55.578491 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:55.578475 5274 scope.go:122] "RemoveContainer" containerID="0b0c8265438d78f5dc0cd70a79231a3c92da53c6b6a03b3199811126e448a055" Oct 31 00:50:55.578773 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:50:55.578720 5274 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b0c8265438d78f5dc0cd70a79231a3c92da53c6b6a03b3199811126e448a055\": container with ID starting with 0b0c8265438d78f5dc0cd70a79231a3c92da53c6b6a03b3199811126e448a055 not found: ID does not exist" containerID="0b0c8265438d78f5dc0cd70a79231a3c92da53c6b6a03b3199811126e448a055" Oct 31 00:50:55.578773 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:55.578750 5274 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b0c8265438d78f5dc0cd70a79231a3c92da53c6b6a03b3199811126e448a055"} err="failed to get container status \"0b0c8265438d78f5dc0cd70a79231a3c92da53c6b6a03b3199811126e448a055\": rpc error: code = NotFound desc = could not find container \"0b0c8265438d78f5dc0cd70a79231a3c92da53c6b6a03b3199811126e448a055\": container with ID starting with 0b0c8265438d78f5dc0cd70a79231a3c92da53c6b6a03b3199811126e448a055 not found: ID does not exist" Oct 31 00:50:55.580983 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:55.580949 5274 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:50:57.480973 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:50:57.480927 5274 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cea6fd7-14ad-45ce-b408-6042192c0a97" path="/var/lib/kubelet/pods/6cea6fd7-14ad-45ce-b408-6042192c0a97/volumes" Oct 31 00:51:20.899375 ip-10-0-129-70 kubenswrapper[5274]: E1031 00:51:20.899334 5274 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-84g8j" containerName="extractor" Oct 31 00:51:24.179475 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:24.179447 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6l8ht_99b4d92d-2003-4708-99da-a5f12cbb10e2/global-pull-secret-syncer/0.log" Oct 31 00:51:24.622925 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:24.622891 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-x962k_409b7296-1d35-44e9-be1a-3459f0dbaad9/konnectivity-agent/0.log" Oct 31 00:51:24.661711 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:24.661674 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-70.us-east-2.compute.internal_2e9a3bb1bd7fbf1e3c59a1731918cc24/haproxy/0.log" Oct 31 00:51:30.343930 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:30.343897 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1ecb2f10-eefd-4680-8741-3f16a479fb2a/alertmanager/0.log" Oct 31 00:51:30.373817 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:30.373783 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1ecb2f10-eefd-4680-8741-3f16a479fb2a/config-reloader/0.log" Oct 31 00:51:30.400422 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:30.400402 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1ecb2f10-eefd-4680-8741-3f16a479fb2a/kube-rbac-proxy-web/0.log" Oct 31 00:51:30.432788 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:30.432767 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1ecb2f10-eefd-4680-8741-3f16a479fb2a/kube-rbac-proxy/0.log" Oct 31 00:51:30.465506 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:30.465483 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1ecb2f10-eefd-4680-8741-3f16a479fb2a/kube-rbac-proxy-metric/0.log" Oct 31 00:51:30.495659 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:30.495643 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1ecb2f10-eefd-4680-8741-3f16a479fb2a/prom-label-proxy/0.log" Oct 31 00:51:30.524344 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:30.524320 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1ecb2f10-eefd-4680-8741-3f16a479fb2a/init-config-reloader/0.log" Oct 31 00:51:30.610436 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:30.610383 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-frfhc_1ad2242c-28d7-4495-9bac-f6c8cba7f796/kube-state-metrics/0.log" Oct 31 00:51:30.635331 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:30.635310 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-frfhc_1ad2242c-28d7-4495-9bac-f6c8cba7f796/kube-rbac-proxy-main/0.log" Oct 31 00:51:30.665229 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:30.665205 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-frfhc_1ad2242c-28d7-4495-9bac-f6c8cba7f796/kube-rbac-proxy-self/0.log" Oct 31 00:51:30.700547 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:30.700528 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6fbfb6ff44-2sxgp_0a7a184e-d0ef-4c2a-ab06-129e37b7e22e/metrics-server/0.log" Oct 31 00:51:30.786007 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:30.785974 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bpczc_de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9/node-exporter/0.log" Oct 31 00:51:30.811098 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:30.811072 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bpczc_de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9/kube-rbac-proxy/0.log" Oct 31 00:51:30.841184 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:30.841163 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bpczc_de1822fa-bfae-4ed1-bf3e-245c0ba3d4b9/init-textfile/0.log" Oct 31 00:51:31.168859 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:31.168832 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-tt54l_5ab2f199-64ea-4071-b8cd-6b9861d31179/kube-rbac-proxy-main/0.log" Oct 31 00:51:31.195469 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:31.195446 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-tt54l_5ab2f199-64ea-4071-b8cd-6b9861d31179/kube-rbac-proxy-self/0.log" Oct 31 00:51:31.225680 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:31.225657 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-tt54l_5ab2f199-64ea-4071-b8cd-6b9861d31179/openshift-state-metrics/0.log" Oct 31 00:51:31.622682 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:31.622647 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5dc5757857-pp2xd_c8d0e1f0-620a-46c0-8234-628fb8acb572/telemeter-client/0.log" Oct 31 00:51:31.646718 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:31.646694 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5dc5757857-pp2xd_c8d0e1f0-620a-46c0-8234-628fb8acb572/reload/0.log" Oct 31 00:51:31.671297 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:31.671270 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5dc5757857-pp2xd_c8d0e1f0-620a-46c0-8234-628fb8acb572/kube-rbac-proxy/0.log" Oct 31 00:51:32.957347 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:32.957317 5274 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xtkzz/perf-node-gather-daemonset-wq8fr"] Oct 31 00:51:32.957718 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:32.957636 5274 cpu_manager.go:414] "containerMap: removing container" podUID="6131e5be-2a37-463e-ad2b-d1f60b831a7e" containerName="kube-rbac-proxy" Oct 31 00:51:32.957718 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:32.957655 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6131e5be-2a37-463e-ad2b-d1f60b831a7e" containerName="kube-rbac-proxy" Oct 31 00:51:32.957718 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:32.957670 5274 cpu_manager.go:414] "containerMap: removing container" podUID="6cea6fd7-14ad-45ce-b408-6042192c0a97" containerName="alertmanager" Oct 31 00:51:32.957718 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:32.957679 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cea6fd7-14ad-45ce-b408-6042192c0a97" containerName="alertmanager" Oct 31 00:51:32.957718 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:32.957699 5274 cpu_manager.go:414] "containerMap: removing container" podUID="6cea6fd7-14ad-45ce-b408-6042192c0a97" containerName="config-reloader" Oct 31 00:51:32.957718 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:32.957704 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cea6fd7-14ad-45ce-b408-6042192c0a97" containerName="config-reloader" Oct 31 00:51:32.957718 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:32.957714 5274 cpu_manager.go:414] "containerMap: removing container" podUID="db1066ed-101f-436d-916e-88fea4540e87" containerName="otc-container" Oct 31 00:51:32.957718 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:32.957719 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db1066ed-101f-436d-916e-88fea4540e87" containerName="otc-container" Oct 31 00:51:32.958059 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:32.957726 5274 cpu_manager.go:414] "containerMap: removing container" podUID="094e5b88-2f67-4ec4-9847-f03dfd7eca5b" containerName="prometheus" Oct 31 00:51:32.958059 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:32.957732 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="094e5b88-2f67-4ec4-9847-f03dfd7eca5b" containerName="prometheus" Oct 31 00:51:32.958059 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:32.957739 5274 cpu_manager.go:414] "containerMap: removing container" podUID="094e5b88-2f67-4ec4-9847-f03dfd7eca5b" containerName="init-config-reloader" Oct 31 00:51:32.958059 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:32.957743 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="094e5b88-2f67-4ec4-9847-f03dfd7eca5b" containerName="init-config-reloader" Oct 31 00:51:32.958059 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:32.957749 5274 cpu_manager.go:414] "containerMap: removing container" podUID="094e5b88-2f67-4ec4-9847-f03dfd7eca5b" containerName="thanos-sidecar" Oct 31 00:51:32.958059 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:32.957753 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="094e5b88-2f67-4ec4-9847-f03dfd7eca5b" containerName="thanos-sidecar" Oct 31 00:51:32.958059 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:32.957761 5274 cpu_manager.go:414] "containerMap: removing container" podUID="6cea6fd7-14ad-45ce-b408-6042192c0a97" containerName="init-config-reloader" Oct 31 00:51:32.958059 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:32.957765 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cea6fd7-14ad-45ce-b408-6042192c0a97" containerName="init-config-reloader" Oct 31 00:51:32.958059 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:32.957773 5274 cpu_manager.go:414] "containerMap: removing container" podUID="094e5b88-2f67-4ec4-9847-f03dfd7eca5b" containerName="config-reloader" Oct 31 00:51:32.958059 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:32.957777 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="094e5b88-2f67-4ec4-9847-f03dfd7eca5b" containerName="config-reloader" Oct 31 00:51:32.958059 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:32.957787 5274 cpu_manager.go:414] "containerMap: removing container" podUID="c56cc3d3-d13b-4b98-a94f-4de0d7d6f796" containerName="korrel8r" Oct 31 00:51:32.958059 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:32.957792 5274 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c56cc3d3-d13b-4b98-a94f-4de0d7d6f796" containerName="korrel8r" Oct 31 00:51:32.958059 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:32.957851 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="6131e5be-2a37-463e-ad2b-d1f60b831a7e" containerName="kube-rbac-proxy" Oct 31 00:51:32.958059 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:32.957860 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cea6fd7-14ad-45ce-b408-6042192c0a97" containerName="alertmanager" Oct 31 00:51:32.958059 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:32.957868 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cea6fd7-14ad-45ce-b408-6042192c0a97" containerName="config-reloader" Oct 31 00:51:32.958059 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:32.957874 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="db1066ed-101f-436d-916e-88fea4540e87" containerName="otc-container" Oct 31 00:51:32.958059 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:32.957880 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="094e5b88-2f67-4ec4-9847-f03dfd7eca5b" containerName="prometheus" Oct 31 00:51:32.958059 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:32.957885 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="094e5b88-2f67-4ec4-9847-f03dfd7eca5b" containerName="thanos-sidecar" Oct 31 00:51:32.958059 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:32.957891 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="094e5b88-2f67-4ec4-9847-f03dfd7eca5b" containerName="config-reloader" Oct 31 00:51:32.958059 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:32.957898 5274 memory_manager.go:365] "RemoveStaleState removing state" podUID="c56cc3d3-d13b-4b98-a94f-4de0d7d6f796" containerName="korrel8r" Oct 31 00:51:32.960663 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:32.960647 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-wq8fr" Oct 31 00:51:32.969876 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:32.969829 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xtkzz\"/\"kube-root-ca.crt\"" Oct 31 00:51:32.970036 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:32.969892 5274 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xtkzz\"/\"default-dockercfg-bmqgh\"" Oct 31 00:51:32.971018 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:32.970996 5274 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xtkzz\"/\"openshift-service-ca.crt\"" Oct 31 00:51:32.972643 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:32.972624 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xtkzz/perf-node-gather-daemonset-wq8fr"] Oct 31 00:51:33.120280 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:33.120256 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5b4455f9-6e7b-4d19-83c9-7655dc5ae6d8-lib-modules\") pod \"perf-node-gather-daemonset-wq8fr\" (UID: \"5b4455f9-6e7b-4d19-83c9-7655dc5ae6d8\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-wq8fr" Oct 31 00:51:33.120425 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:33.120290 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5b4455f9-6e7b-4d19-83c9-7655dc5ae6d8-podres\") pod \"perf-node-gather-daemonset-wq8fr\" (UID: \"5b4455f9-6e7b-4d19-83c9-7655dc5ae6d8\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-wq8fr" Oct 31 00:51:33.120425 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:33.120391 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b4455f9-6e7b-4d19-83c9-7655dc5ae6d8-sys\") pod \"perf-node-gather-daemonset-wq8fr\" (UID: \"5b4455f9-6e7b-4d19-83c9-7655dc5ae6d8\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-wq8fr" Oct 31 00:51:33.120522 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:33.120437 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5b4455f9-6e7b-4d19-83c9-7655dc5ae6d8-proc\") pod \"perf-node-gather-daemonset-wq8fr\" (UID: \"5b4455f9-6e7b-4d19-83c9-7655dc5ae6d8\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-wq8fr" Oct 31 00:51:33.120522 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:33.120489 5274 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skp8q\" (UniqueName: \"kubernetes.io/projected/5b4455f9-6e7b-4d19-83c9-7655dc5ae6d8-kube-api-access-skp8q\") pod \"perf-node-gather-daemonset-wq8fr\" (UID: \"5b4455f9-6e7b-4d19-83c9-7655dc5ae6d8\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-wq8fr" Oct 31 00:51:33.221543 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:33.221516 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5b4455f9-6e7b-4d19-83c9-7655dc5ae6d8-lib-modules\") pod \"perf-node-gather-daemonset-wq8fr\" (UID: \"5b4455f9-6e7b-4d19-83c9-7655dc5ae6d8\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-wq8fr" Oct 31 00:51:33.221543 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:33.221547 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5b4455f9-6e7b-4d19-83c9-7655dc5ae6d8-podres\") pod \"perf-node-gather-daemonset-wq8fr\" (UID: \"5b4455f9-6e7b-4d19-83c9-7655dc5ae6d8\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-wq8fr" Oct 31 00:51:33.221726 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:33.221569 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b4455f9-6e7b-4d19-83c9-7655dc5ae6d8-sys\") pod \"perf-node-gather-daemonset-wq8fr\" (UID: \"5b4455f9-6e7b-4d19-83c9-7655dc5ae6d8\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-wq8fr" Oct 31 00:51:33.221726 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:33.221600 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5b4455f9-6e7b-4d19-83c9-7655dc5ae6d8-proc\") pod \"perf-node-gather-daemonset-wq8fr\" (UID: \"5b4455f9-6e7b-4d19-83c9-7655dc5ae6d8\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-wq8fr" Oct 31 00:51:33.221726 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:33.221629 5274 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skp8q\" (UniqueName: \"kubernetes.io/projected/5b4455f9-6e7b-4d19-83c9-7655dc5ae6d8-kube-api-access-skp8q\") pod \"perf-node-gather-daemonset-wq8fr\" (UID: \"5b4455f9-6e7b-4d19-83c9-7655dc5ae6d8\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-wq8fr" Oct 31 00:51:33.221726 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:33.221651 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5b4455f9-6e7b-4d19-83c9-7655dc5ae6d8-lib-modules\") pod \"perf-node-gather-daemonset-wq8fr\" (UID: \"5b4455f9-6e7b-4d19-83c9-7655dc5ae6d8\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-wq8fr" Oct 31 00:51:33.221726 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:33.221654 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5b4455f9-6e7b-4d19-83c9-7655dc5ae6d8-podres\") pod \"perf-node-gather-daemonset-wq8fr\" (UID: \"5b4455f9-6e7b-4d19-83c9-7655dc5ae6d8\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-wq8fr" Oct 31 00:51:33.221726 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:33.221658 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b4455f9-6e7b-4d19-83c9-7655dc5ae6d8-sys\") pod \"perf-node-gather-daemonset-wq8fr\" (UID: \"5b4455f9-6e7b-4d19-83c9-7655dc5ae6d8\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-wq8fr" Oct 31 00:51:33.221726 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:33.221694 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5b4455f9-6e7b-4d19-83c9-7655dc5ae6d8-proc\") pod \"perf-node-gather-daemonset-wq8fr\" (UID: \"5b4455f9-6e7b-4d19-83c9-7655dc5ae6d8\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-wq8fr" Oct 31 00:51:33.233980 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:33.233944 5274 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skp8q\" (UniqueName: \"kubernetes.io/projected/5b4455f9-6e7b-4d19-83c9-7655dc5ae6d8-kube-api-access-skp8q\") pod \"perf-node-gather-daemonset-wq8fr\" (UID: \"5b4455f9-6e7b-4d19-83c9-7655dc5ae6d8\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-wq8fr" Oct 31 00:51:33.273081 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:33.273060 5274 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-wq8fr" Oct 31 00:51:33.543558 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:33.543535 5274 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xtkzz/perf-node-gather-daemonset-wq8fr"] Oct 31 00:51:33.545736 ip-10-0-129-70 kubenswrapper[5274]: W1031 00:51:33.545707 5274 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5b4455f9_6e7b_4d19_83c9_7655dc5ae6d8.slice/crio-c41bfa1baa7028359778545f3f081e11f34965835182474bdd9926f7ee741600 WatchSource:0}: Error finding container c41bfa1baa7028359778545f3f081e11f34965835182474bdd9926f7ee741600: Status 404 returned error can't find the container with id c41bfa1baa7028359778545f3f081e11f34965835182474bdd9926f7ee741600 Oct 31 00:51:33.753211 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:33.753175 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-wq8fr" event={"ID":"5b4455f9-6e7b-4d19-83c9-7655dc5ae6d8","Type":"ContainerStarted","Data":"8ca78e88af3cc516f06ed22d5751dc8d08506aeabb98ab8fa723e9e2f6bc7e90"} Oct 31 00:51:33.753365 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:33.753226 5274 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-wq8fr" event={"ID":"5b4455f9-6e7b-4d19-83c9-7655dc5ae6d8","Type":"ContainerStarted","Data":"c41bfa1baa7028359778545f3f081e11f34965835182474bdd9926f7ee741600"} Oct 31 00:51:33.753365 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:33.753267 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-wq8fr" Oct 31 00:51:33.771946 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:33.771895 5274 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-wq8fr" podStartSLOduration=1.771883157 podStartE2EDuration="1.771883157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:51:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:33.770251674 +0000 UTC m=+3130.407070393" watchObservedRunningTime="2026-10-31 00:51:33.771883157 +0000 UTC m=+3130.408701873" Oct 31 00:51:36.326527 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:36.326497 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-g5p2k_8b099593-ded8-475c-b5ef-a91bbd9aeb1a/dns/0.log" Oct 31 00:51:36.349819 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:36.349795 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-g5p2k_8b099593-ded8-475c-b5ef-a91bbd9aeb1a/kube-rbac-proxy/0.log" Oct 31 00:51:36.429682 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:36.429659 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6hq7b_d7815306-7a99-4fc5-8226-b34916ceaaa2/dns-node-resolver/0.log" Oct 31 00:51:37.031412 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:37.031385 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-pruner-29890080-6f25s_3401aa52-8809-45ca-9e99-02fc01a6639d/image-pruner/0.log" Oct 31 00:51:37.155739 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:37.155682 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vnk6j_18acd0a8-988e-4168-84e7-0b25dc7b32d4/node-ca/0.log" Oct 31 00:51:38.394102 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:38.394066 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-djtcv_18698607-3254-4ec9-b3bb-bbe59c6c744f/serve-healthcheck-canary/0.log" Oct 31 00:51:38.836530 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:38.836452 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-84g8j_3cb65abb-ba0e-4075-9a0a-455e1b64b820/kube-rbac-proxy/0.log" Oct 31 00:51:38.859161 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:38.859138 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-84g8j_3cb65abb-ba0e-4075-9a0a-455e1b64b820/exporter/0.log" Oct 31 00:51:38.882587 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:38.882556 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-84g8j_3cb65abb-ba0e-4075-9a0a-455e1b64b820/extractor/0.log" Oct 31 00:51:39.768001 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:39.767975 5274 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-wq8fr" Oct 31 00:51:41.662910 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:41.662878 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-opentelemetry-operator_opentelemetry-operator-controller-manager-5f695b7fbf-p6g75_2b4eaad8-f26b-4848-a9b9-f750079c11e0/manager/0.log" Oct 31 00:51:41.694968 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:41.694924 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-opentelemetry-operator_opentelemetry-operator-webhook-6c7d65b66-cxcqb_c86982f7-6d7d-463d-a107-033c44b29306/webhook/0.log" Oct 31 00:51:42.250205 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:42.250177 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-observability-operator_obo-prometheus-operator-admission-webhook-7fb7b74868-gwkrc_900d5ad4-111a-4a47-9809-643ab8000284/prometheus-operator-admission-webhook/0.log" Oct 31 00:51:42.294141 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:42.294115 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-observability-operator_observability-operator-599c6f89d-bd595_f811b29c-8848-4fc6-a28c-91a034a55c0f/operator/0.log" Oct 31 00:51:42.466373 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:42.466344 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-observability-operator_perses-operator-7c856d4d4c-x67rf_4d969e19-4f85-4e6f-9c5a-e373c53a7bde/perses-operator/0.log" Oct 31 00:51:48.043631 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:48.043592 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9j82q_a0be3213-788c-45b4-8fbd-37e744a14f17/kube-multus/0.log" Oct 31 00:51:48.427518 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:48.427492 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tc87d_d8089437-f546-441e-b121-6e79878d2608/kube-multus-additional-cni-plugins/0.log" Oct 31 00:51:48.451483 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:48.451454 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tc87d_d8089437-f546-441e-b121-6e79878d2608/egress-router-binary-copy/0.log" Oct 31 00:51:48.480678 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:48.480655 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tc87d_d8089437-f546-441e-b121-6e79878d2608/cni-plugins/0.log" Oct 31 00:51:48.505778 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:48.505755 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tc87d_d8089437-f546-441e-b121-6e79878d2608/bond-cni-plugin/0.log" Oct 31 00:51:48.530119 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:48.530101 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tc87d_d8089437-f546-441e-b121-6e79878d2608/routeoverride-cni/0.log" Oct 31 00:51:48.554459 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:48.554424 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tc87d_d8089437-f546-441e-b121-6e79878d2608/whereabouts-cni-bincopy/0.log" Oct 31 00:51:48.582211 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:48.582181 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tc87d_d8089437-f546-441e-b121-6e79878d2608/whereabouts-cni/0.log" Oct 31 00:51:48.854323 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:48.854244 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-cjqfw_0adc098d-57a4-490e-89b7-b1b271a1c087/network-metrics-daemon/0.log" Oct 31 00:51:48.874849 ip-10-0-129-70 kubenswrapper[5274]: I1031 00:51:48.874815 5274 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-cjqfw_0adc098d-57a4-490e-89b7-b1b271a1c087/kube-rbac-proxy/0.log"